2025-12-17 05:33:25 [scrapy.utils.log] INFO: Scrapy 2.11.1 started (bot: news_scraper)
2025-12-17 05:33:25 [scrapy.utils.log] INFO: Versions: lxml 6.0.2.0, libxml2 2.14.6, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 25.5.0, Python 3.11.13 (main, Aug 12 2025, 22:39:41) [GCC 14.2.0], pyOpenSSL 25.3.0 (OpenSSL 3.5.3 16 Sep 2025), cryptography 46.0.1, Platform Linux-5.15.0-157-generic-x86_64-with
2025-12-17 05:33:25 [scrapy.addons] INFO: Enabled addons:
[]
2025-12-17 05:33:25 [asyncio] DEBUG: Using selector: EpollSelector
2025-12-17 05:33:25 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2025-12-17 05:33:25 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2025-12-17 05:33:25 [scrapy.extensions.telnet] INFO: Telnet Password: 30b7ea395085d6d8
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/endpoints.json
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f0a09b3bec0>
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json.gz
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/partitions.json
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f0a09c90a40>
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f0a09a70860>
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f0a09c907c0>
2025-12-17 05:33:25 [botocore.endpoint] DEBUG: Setting s3 timeout as (60, 60)
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/_retry.json
2025-12-17 05:33:25 [botocore.client] DEBUG: Registering retry handlers for service: s3
2025-12-17 05:33:25 [botocore.utils] DEBUG: Registering S3 region redirector handler
2025-12-17 05:33:25 [botocore.utils] DEBUG: Registering S3Express Identity Resolver
2025-12-17 05:33:25 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle']
2025-12-17 05:33:25 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
 'BOT_NAME': 'news_scraper',
 'CLOSESPIDER_TIMEOUT': 1800,
 'CONCURRENT_REQUESTS': 4,
 'DOWNLOAD_DELAY': 2,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/opt/scrapyd/logs/news_scraper/yahooworld_timestamp/d5f011c4db0911f099e2d6783c969646.log',
 'NEWSPIDER_MODULE': 'news_scraper.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['news_scraper.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2025-12-17 05:33:25 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'news_scraper.middlewares.NewsScraperDownloaderMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2025-12-17 05:33:25 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-12-17 05:33:25 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2025-12-17 05:33:25 [scrapy.core.engine] INFO: Spider opened
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/endpoints.json
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2025-12-17 05:33:25 [botocore.hooks] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f0a09b3bec0>
2025-12-17 05:33:25 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json.gz
2025-12-17 05:33:26 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2025-12-17 05:33:26 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/partitions.json
2025-12-17 05:33:26 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f0a09c90a40>
2025-12-17 05:33:26 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f0a08ad2de0>
2025-12-17 05:33:26 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f0a09c907c0>
2025-12-17 05:33:26 [botocore.endpoint] DEBUG: Setting s3 timeout as (60, 60)
2025-12-17 05:33:26 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/_retry.json
2025-12-17 05:33:26 [botocore.client] DEBUG: Registering retry handlers for service: s3
2025-12-17 05:33:26 [botocore.utils] DEBUG: Registering S3 region redirector handler
2025-12-17 05:33:26 [botocore.utils] DEBUG: Registering S3Express Identity Resolver
2025-12-17 05:33:26 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-12-17 05:33:26 [yahooworld_timestamp] INFO: Spider opened: yahooworld_timestamp
2025-12-17 05:33:26 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6029
2025-12-17 05:33:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.yahoo.com/> (referer: None)
2025-12-17 05:33:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.yahoo.com/world/> (referer: https://news.yahoo.com/)
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/boxing/article/shakur-stevenson-admits-sparring-jake-paul-was-eye-opening-hes-better-than-people-would-even-understand-anthony-joshua-221751447.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/fantasy/article/fantasy-football-week-16-running-back-rankings-182022925.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/college-football/article/college-football-playoff-picks-predictions-for-2025-2026-bracket-who-will-win-it-all-184923686.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/personal-finance/mortgages/article/mortgage-refinance-rates-today-tuesday-december-16-2025-110035487.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/nba/article/kings-could-look-to-trade-veteran-players-due-to-reported-disconnect-with-coaching-staff-173145409.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://health.yahoo.com/conditions/mental-health/article/no-one-died-so-why-does-it-hurt-this-much-experts-explain-7-types-of-grief-043409638.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/mma/article/ufc-schedule-fight-cards-start-times-odds-how-to-watch-212552711.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/live/trump-tariffs-live-updates-us-suspends-tech-deal-with-uk-trump-has-said-tariff-revenues-could-pay-for-at-least-9-different-things-231853050.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://health.yahoo.com/wellness/article/i-was-36-when-my-husband-died--heres-what-most-of-us-get-wrong-about-grief-193742038.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/us/article/manhunt-for-brown-university-shooter-continues-fbi-releases-photos-of-suspect-announces-50k-reward-153638968.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/live/rob-reiners-son-nick-arrested-on-murder-charges-after-director-and-his-wife-found-dead-at-their-la-home-follow-live-updates-135653863.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/the-us-labor-market-ground-to-a-halt-in-2025-the-risk-in-2026-is-that-it-cracks-140026614.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://tech.yahoo.com/puzzles/wordle/article/wordle-hints-today-for-1642-clues-and-answer-for-wednesday-december-17-050111902.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://tech.yahoo.com/puzzles/strands/article/nyt-strands-hints-today-for-654-clues-and-answers-for-wednesday-dec-17-2025-050141831.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/nba/article/ime-udoka-blasts-refs-after-overtime-loss-to-nuggets-most-poorly-officiated-game-ive-seen-in-a-long-time-133259503.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://ca.news.yahoo.com/tell-me-softly-top-prime-video-film-is-another-spanish-language-romance-drama-with-a-sibling-love-triangle-042958252.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sg.news.yahoo.com/year-in-review-2025-top-5-sports-highlights-025010784.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/movies/article/the-movie-thats-making-theatergoers-sob-uncontrollably-200006458.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://tech.yahoo.com/audio/deals/article/apple-headphones-dont-get-cheaper-than-this-earpods-are-down-to-just-11-200052019.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/us/article/who-is-nick-reiner-son-of-rob-michele-reiner-booked-for-murder-in-parents-deaths-172649454.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/live/stock-market-today-dow-sp-500-slip-nasdaq-snaps-three-day-losing-streak-as-tesla-climbs-to-record-210029709.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.engadget.com/transportation/evs/tesla-used-deceptive-language-to-market-autopilot-california-judge-rules-035826786.html
2025-12-17 05:33:33 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.engadget.com': <GET https://www.engadget.com/transportation/evs/tesla-used-deceptive-language-to-market-autopilot-california-judge-rules-035826786.html>
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/us/article/a-new-flu-variant-is-creating-surging-cases-earlier-than-expected-and-causing-severe-illness-experts-warn-what-you-need-to-know-001640989.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/articles/nick-reiner-reportedly-stormed-off-224732845.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/articles/jason-ritter-reflects-childhood-anger-002700694.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/trump-very-strongly-considering-major-175915515.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/lifestyle/articles/repairman-breaks-open-washing-machine-090000789.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/articles/moment-trump-goons-realized-vanity-181520735.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/articles/longtime-sports-reporter-found-dead-194351274.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/people-noticing-something-very-weird-203108677.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/michelle-obama-reacts-trump-rob-114054986.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/tv/articles/jimmy-fallon-under-fire-over-125944063.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/jd-vance-fires-back-says-182148046.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/articles/britney-spears-suffers-wardrobe-malfunction-163017646.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/movies/articles/avatar-fire-ash-review-first-140000323.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/michael-saylor-buys-another-1b-121215863.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/daughter-hong-kong-tycoon-jimmy-043409521.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/powerball-winner-sold-watkins-glen-153836507.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/pumas-came-back-patagonia-met-000100387.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/articles/doctor-problem-chiefs-patrick-mahomes-213010442.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/articles/onlyfans-sophie-rain-posing-mini-135124697.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/big-wind-event-colorados-xcel-001637109.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/mysterious-bright-flashes-night-sky-110000813.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://health.yahoo.com/wellness/articles/eventually-kill-medical-professionals-revealing-173103773.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/archaeologists-pried-open-living-room-140000093.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/lifestyle/articles/labrador-extra-mushy-morning-owner-132102313.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/jim-cramer-says-time-sell-213044796.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/articles/michigan-announces-final-decision-woman-144646650.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/aftermath-ukraine-underwater-drone-attack-231955075.html
2025-12-17 05:33:33 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/articles/flabbergasted-troy-aikman-did-not-062821109.html
2025-12-17 05:33:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/boxing/article/shakur-stevenson-admits-sparring-jake-paul-was-eye-opening-hes-better-than-people-would-even-understand-anthony-joshua-221751447.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:34 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/boxing/article/shakur-stevenson-admits-sparring-jake-paul-was-eye-opening-hes-better-than-people-would-even-understand-anthony-joshua-221751447.html
2025-12-17 05:33:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://health.yahoo.com/wellness/article/i-was-36-when-my-husband-died--heres-what-most-of-us-get-wrong-about-grief-193742038.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/live/trump-tariffs-live-updates-us-suspends-tech-deal-with-uk-trump-has-said-tariff-revenues-could-pay-for-at-least-9-different-things-231853050.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:35 [yahooworld_timestamp] INFO: 2025-12-17 02:37:42 smaller than 2025-12-17 12:17:00
2025-12-17 05:33:35 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/live/trump-tariffs-live-updates-us-suspends-tech-deal-with-uk-trump-has-said-tariff-revenues-could-pay-for-at-least-9-different-things-231853050.html
2025-12-17 05:33:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/aftermath-ukraine-underwater-drone-attack-231955075.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:37 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/aftermath-ukraine-underwater-drone-attack-231955075.html
2025-12-17 05:33:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/jim-cramer-says-time-sell-213044796.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:38 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/jim-cramer-says-time-sell-213044796.html
2025-12-17 05:33:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/mma/article/ufc-schedule-fight-cards-start-times-odds-how-to-watch-212552711.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:39 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/mma/article/ufc-schedule-fight-cards-start-times-odds-how-to-watch-212552711.html
2025-12-17 05:33:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/lifestyle/articles/labrador-extra-mushy-morning-owner-132102313.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:40 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/lifestyle/articles/labrador-extra-mushy-morning-owner-132102313.html
2025-12-17 05:33:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/archaeologists-pried-open-living-room-140000093.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:40 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/archaeologists-pried-open-living-room-140000093.html
2025-12-17 05:33:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://health.yahoo.com/wellness/articles/eventually-kill-medical-professionals-revealing-173103773.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:41 [yahooworld_timestamp] INFO: 2025-12-17 00:31:03 smaller than 2025-12-17 12:17:00
2025-12-17 05:33:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/articles/flabbergasted-troy-aikman-did-not-062821109.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:43 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/articles/flabbergasted-troy-aikman-did-not-062821109.html
2025-12-17 05:33:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/mysterious-bright-flashes-night-sky-110000813.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:44 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/mysterious-bright-flashes-night-sky-110000813.html
2025-12-17 05:33:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/articles/michigan-announces-final-decision-woman-144646650.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:45 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/articles/michigan-announces-final-decision-woman-144646650.html
2025-12-17 05:33:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/big-wind-event-colorados-xcel-001637109.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:46 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/big-wind-event-colorados-xcel-001637109.html
2025-12-17 05:33:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/articles/doctor-problem-chiefs-patrick-mahomes-213010442.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:48 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/articles/doctor-problem-chiefs-patrick-mahomes-213010442.html
2025-12-17 05:33:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/articles/onlyfans-sophie-rain-posing-mini-135124697.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:49 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/articles/onlyfans-sophie-rain-posing-mini-135124697.html
2025-12-17 05:33:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/michael-saylor-buys-another-1b-121215863.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:50 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/michael-saylor-buys-another-1b-121215863.html
2025-12-17 05:33:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/pumas-came-back-patagonia-met-000100387.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:51 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/pumas-came-back-patagonia-met-000100387.html
2025-12-17 05:33:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/powerball-winner-sold-watkins-glen-153836507.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:54 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/powerball-winner-sold-watkins-glen-153836507.html
2025-12-17 05:33:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/daughter-hong-kong-tycoon-jimmy-043409521.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:55 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/daughter-hong-kong-tycoon-jimmy-043409521.html
2025-12-17 05:33:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/movies/articles/avatar-fire-ash-review-first-140000323.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:33:58 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/movies/articles/avatar-fire-ash-review-first-140000323.html
2025-12-17 05:34:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/articles/britney-spears-suffers-wardrobe-malfunction-163017646.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:01 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/articles/britney-spears-suffers-wardrobe-malfunction-163017646.html
2025-12-17 05:34:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/jd-vance-fires-back-says-182148046.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:02 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/jd-vance-fires-back-says-182148046.html
2025-12-17 05:34:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/articles/longtime-sports-reporter-found-dead-194351274.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:02 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/articles/longtime-sports-reporter-found-dead-194351274.html
2025-12-17 05:34:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/tv/articles/jimmy-fallon-under-fire-over-125944063.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:05 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/tv/articles/jimmy-fallon-under-fire-over-125944063.html
2025-12-17 05:34:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/michelle-obama-reacts-trump-rob-114054986.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:08 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/michelle-obama-reacts-trump-rob-114054986.html
2025-12-17 05:34:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/people-noticing-something-very-weird-203108677.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:10 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/people-noticing-something-very-weird-203108677.html
2025-12-17 05:34:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/articles/moment-trump-goons-realized-vanity-181520735.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:12 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/articles/moment-trump-goons-realized-vanity-181520735.html
2025-12-17 05:34:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/lifestyle/articles/repairman-breaks-open-washing-machine-090000789.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:14 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/lifestyle/articles/repairman-breaks-open-washing-machine-090000789.html
2025-12-17 05:34:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/trump-very-strongly-considering-major-175915515.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:17 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/trump-very-strongly-considering-major-175915515.html
2025-12-17 05:34:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/live/stock-market-today-dow-sp-500-slip-nasdaq-snaps-three-day-losing-streak-as-tesla-climbs-to-record-210029709.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:18 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/live/stock-market-today-dow-sp-500-slip-nasdaq-snaps-three-day-losing-streak-as-tesla-climbs-to-record-210029709.html
2025-12-17 05:34:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/articles/jason-ritter-reflects-childhood-anger-002700694.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:19 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/articles/jason-ritter-reflects-childhood-anger-002700694.html
2025-12-17 05:34:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tech.yahoo.com/audio/deals/article/apple-headphones-dont-get-cheaper-than-this-earpods-are-down-to-just-11-200052019.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:20 [yahooworld_timestamp] INFO: 2025-12-17 03:00:52 smaller than 2025-12-17 12:17:00
2025-12-17 05:34:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/articles/nick-reiner-reportedly-stormed-off-224732845.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:22 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/articles/nick-reiner-reportedly-stormed-off-224732845.html
2025-12-17 05:34:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sg.news.yahoo.com/year-in-review-2025-top-5-sports-highlights-025010784.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:22 [yahooworld_timestamp] INFO: Invalid article: https://sg.news.yahoo.com/year-in-review-2025-top-5-sports-highlights-025010784.html
2025-12-17 05:34:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://ca.news.yahoo.com/tell-me-softly-top-prime-video-film-is-another-spanish-language-romance-drama-with-a-sibling-love-triangle-042958252.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:23 [yahooworld_timestamp] INFO: Invalid article: https://ca.news.yahoo.com/tell-me-softly-top-prime-video-film-is-another-spanish-language-romance-drama-with-a-sibling-love-triangle-042958252.html
2025-12-17 05:34:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/nba/article/ime-udoka-blasts-refs-after-overtime-loss-to-nuggets-most-poorly-officiated-game-ive-seen-in-a-long-time-133259503.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/us/article/a-new-flu-variant-is-creating-surging-cases-earlier-than-expected-and-causing-severe-illness-experts-warn-what-you-need-to-know-001640989.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:23 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/nba/article/ime-udoka-blasts-refs-after-overtime-loss-to-nuggets-most-poorly-officiated-game-ive-seen-in-a-long-time-133259503.html
2025-12-17 05:34:24 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/us/article/a-new-flu-variant-is-creating-surging-cases-earlier-than-expected-and-causing-severe-illness-experts-warn-what-you-need-to-know-001640989.html
2025-12-17 05:34:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tech.yahoo.com/puzzles/strands/article/nyt-strands-hints-today-for-654-clues-and-answers-for-wednesday-dec-17-2025-050141831.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:24 [yahooworld_timestamp] INFO: 2025-12-17 12:01:41 smaller than 2025-12-17 12:17:00
2025-12-17 05:34:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/the-us-labor-market-ground-to-a-halt-in-2025-the-risk-in-2026-is-that-it-cracks-140026614.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/us/article/who-is-nick-reiner-son-of-rob-michele-reiner-booked-for-murder-in-parents-deaths-172649454.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:25 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/the-us-labor-market-ground-to-a-halt-in-2025-the-risk-in-2026-is-that-it-cracks-140026614.html
2025-12-17 05:34:26 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/us/article/who-is-nick-reiner-son-of-rob-michele-reiner-booked-for-murder-in-parents-deaths-172649454.html
2025-12-17 05:34:26 [scrapy.extensions.logstats] INFO: Crawled 42 pages (at 42 pages/min), scraped 0 items (at 0 items/min)
2025-12-17 05:34:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tech.yahoo.com/puzzles/wordle/article/wordle-hints-today-for-1642-clues-and-answer-for-wednesday-december-17-050111902.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:27 [yahooworld_timestamp] INFO: 2025-12-17 12:01:11 smaller than 2025-12-17 12:17:00
2025-12-17 05:34:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://health.yahoo.com/conditions/mental-health/article/no-one-died-so-why-does-it-hurt-this-much-experts-explain-7-types-of-grief-043409638.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:28 [yahooworld_timestamp] INFO: 2025-12-17 11:34:09 smaller than 2025-12-17 12:17:00
2025-12-17 05:34:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/movies/article/the-movie-thats-making-theatergoers-sob-uncontrollably-200006458.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:29 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/movies/article/the-movie-thats-making-theatergoers-sob-uncontrollably-200006458.html
2025-12-17 05:34:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/nba/article/kings-could-look-to-trade-veteran-players-due-to-reported-disconnect-with-coaching-staff-173145409.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:29 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/nba/article/kings-could-look-to-trade-veteran-players-due-to-reported-disconnect-with-coaching-staff-173145409.html
2025-12-17 05:34:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/personal-finance/mortgages/article/mortgage-refinance-rates-today-tuesday-december-16-2025-110035487.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:30 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/personal-finance/mortgages/article/mortgage-refinance-rates-today-tuesday-december-16-2025-110035487.html
2025-12-17 05:34:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/college-football/article/college-football-playoff-picks-predictions-for-2025-2026-bracket-who-will-win-it-all-184923686.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:30 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/college-football/article/college-football-playoff-picks-predictions-for-2025-2026-bracket-who-will-win-it-all-184923686.html
2025-12-17 05:34:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/live/rob-reiners-son-nick-arrested-on-murder-charges-after-director-and-his-wife-found-dead-at-their-la-home-follow-live-updates-135653863.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:31 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/live/rob-reiners-son-nick-arrested-on-murder-charges-after-director-and-his-wife-found-dead-at-their-la-home-follow-live-updates-135653863.html
2025-12-17 05:34:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/us/article/manhunt-for-brown-university-shooter-continues-fbi-releases-photos-of-suspect-announces-50k-reward-153638968.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/fantasy/article/fantasy-football-week-16-running-back-rankings-182022925.html> (referer: https://news.yahoo.com/world/)
2025-12-17 05:34:33 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/us/article/manhunt-for-brown-university-shooter-continues-fbi-releases-photos-of-suspect-announces-50k-reward-153638968.html
2025-12-17 05:34:33 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/fantasy/article/fantasy-football-week-16-running-back-rankings-182022925.html
2025-12-17 05:34:33 [scrapy.core.engine] INFO: Closing spider (finished)
2025-12-17 05:34:33 [boto3.s3.transfer] DEBUG: Opting out of CRT Transfer Manager. Preferred client: auto, CRT available: False, Instance Optimized: False.
2025-12-17 05:34:33 [boto3.s3.transfer] DEBUG: Using default client. pid: 189327, thread: 139681053674296
2025-12-17 05:34:33 [s3transfer.utils] DEBUG: Acquiring 0
2025-12-17 05:34:33 [s3transfer.tasks] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f0a08260e90>}) about to wait for the following futures []
2025-12-17 05:34:33 [s3transfer.tasks] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f0a08260e90>}) done waiting for dependent futures
2025-12-17 05:34:33 [s3transfer.tasks] DEBUG: Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f0a08260e90>}) with kwargs {'client': <botocore.client.S3 object at 0x7f0a09336910>, 'config': <boto3.s3.transfer.TransferConfig object at 0x7f0a082c8210>, 'osutil': <s3transfer.utils.OSUtils object at 0x7f0a082c9dd0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7f0a08261710>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f0a08260e90>}
2025-12-17 05:34:33 [s3transfer.futures] DEBUG: Submitting task PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0a08261710> for transfer request: 0.
2025-12-17 05:34:33 [s3transfer.utils] DEBUG: Acquiring 0
2025-12-17 05:34:33 [s3transfer.tasks] DEBUG: PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'extra_args': {}}) about to wait for the following futures []
2025-12-17 05:34:33 [s3transfer.tasks] DEBUG: PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'extra_args': {}}) done waiting for dependent futures
2025-12-17 05:34:33 [s3transfer.tasks] DEBUG: Executing task PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7f0a09336910>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x7f0a0828f210>, 'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'extra_args': {}}
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x7f0a09b4e660>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x7f0a09b4d940>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x7f0a09b4f060>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x7f0a09b4d8a0>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f0a09b4fa60>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f0a095e3f50>>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x7f0a096208d0>>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x7f0a09b4d6c0>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f0a09b4fc40>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f0a095e3f50>>
2025-12-17 05:34:33 [botocore.regions] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'dagster-output-data', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://lake-api.actable.ai/', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2025-12-17 05:34:33 [botocore.regions] DEBUG: Endpoint provider result: https://lake-api.actable.ai/dagster-output-data
2025-12-17 05:34:33 [botocore.regions] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2025-12-17 05:34:33 [botocore.regions] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function conditionally_calculate_checksum at 0x7f0a09fb4d60>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x7f0a09b4dc60>
2025-12-17 05:34:33 [botocore.handlers] DEBUG: Adding expect 100 continue header to request.
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f0a0969fad0>>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function add_recursion_detection_header at 0x7f0a09b3bf60>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x7f0a09b4f1a0>
2025-12-17 05:34:33 [botocore.endpoint] DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.57 md/Botocore#1.34.162 ua/2.0 os/linux#5.15.0-157-generic md/arch#x86_64 lang/python#3.11.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.162', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7f0a0828f210>, 'auth_path': '/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'url': 'https://lake-api.actable.ai/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f0a09364fd0>, 'has_streaming_input': True, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'dagster-output-data', 'params': {'Bucket': 'dagster-output-data', 'Key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl', 'Body': <s3transfer.utils.ReadFileChunk object at 0x7f0a0828f210>}}, 'input_params': {'Bucket': 'dagster-output-data', 'Key': 'yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl'}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function signal_not_transferring at 0x7f0a08e600e0>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f0a09364f90>>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f0a096208d0>>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7f0a09b4d580>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x7f0a09b4fba0>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event before-sign.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f0a0969fad0>>
2025-12-17 05:34:33 [botocore.auth] DEBUG: Calculating signature using v4 auth.
2025-12-17 05:34:33 [botocore.auth] DEBUG: CanonicalRequest:
PUT
/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl

content-md5:1B2M2Y8AsgTpgAmY7PhCfg==
host:lake-api.actable.ai
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20251217T053433Z

content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2025-12-17 05:34:33 [botocore.auth] DEBUG: StringToSign:
AWS4-HMAC-SHA256
20251217T053433Z
20251217/us-east-1/s3/aws4_request
aa16641c8e1d9032b7cecf5cfe01159492911b4294931b61670330e300e05116
2025-12-17 05:34:33 [botocore.auth] DEBUG: Signature:
c64e70a9d75ea41cb28c4e9cbb4e6566f69ad07bc3003cd884f9003c0c1b27fe
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function signal_transferring at 0x7f0a08e60180>
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7f0a09b4f9c0>
2025-12-17 05:34:33 [botocore.endpoint] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lake-api.actable.ai/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl, headers={'User-Agent': b'Boto3/1.34.57 md/Botocore#1.34.162 ua/2.0 os/linux#5.15.0-157-generic md/arch#x86_64 lang/python#3.11.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.162', 'Content-MD5': b'1B2M2Y8AsgTpgAmY7PhCfg==', 'Expect': b'100-continue', 'X-Amz-Date': b'20251217T053433Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=9zTeMQWwQiGhbKvMJ7uq/20251217/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=c64e70a9d75ea41cb28c4e9cbb4e6566f69ad07bc3003cd884f9003c0c1b27fe', 'amz-sdk-invocation-id': b'f04e1418-0710-4433-9bda-f41c119e69ab', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
2025-12-17 05:34:33 [botocore.httpsession] DEBUG: Certificate path: /usr/local/lib/python3.11/site-packages/certifi/cacert.pem
2025-12-17 05:34:33 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): lake-api.actable.ai:443
2025-12-17 05:34:33 [s3transfer.utils] DEBUG: Releasing acquire 0/None
2025-12-17 05:34:33 [botocore.awsrequest] DEBUG: Waiting for 100 Continue response.
2025-12-17 05:34:33 [botocore.awsrequest] DEBUG: 100 Continue response seen, now sending request body.
2025-12-17 05:34:33 [urllib3.connectionpool] DEBUG: https://lake-api.actable.ai:443 "PUT /dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl HTTP/1.1" 200 0
2025-12-17 05:34:33 [botocore.parsers] DEBUG: Response headers: {'Server': 'nginx/1.24.0 (Ubuntu)', 'Date': 'Wed, 17 Dec 2025 05:34:33 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Accept-Ranges': 'bytes', 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Bucket-Region': 'us-east-1', 'X-Amz-Id-2': 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8', 'X-Amz-Request-Id': '1881E9DF246C7F8A', 'X-Content-Type-Options': 'nosniff', 'X-Ratelimit-Limit': '25637', 'X-Ratelimit-Remaining': '25637', 'X-Xss-Protection': '1; mode=block'}
2025-12-17 05:34:33 [botocore.parsers] DEBUG: Response body:
b''
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f0a0969f990>
2025-12-17 05:34:33 [botocore.retryhandler] DEBUG: No retry needed.
2025-12-17 05:34:33 [botocore.hooks] DEBUG: Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f0a095e3f50>>
2025-12-17 05:34:33 [s3transfer.utils] DEBUG: Releasing acquire 0/None
2025-12-17 05:34:33 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (0 items) in: s3://dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_d5f011c4db0911f099e2d6783c969646_scheduled_2025-12-17.jl
2025-12-17 05:34:33 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 48003,
 'downloader/request_count': 51,
 'downloader/request_method_count/GET': 51,
 'downloader/response_bytes': 8412528,
 'downloader/response_count': 51,
 'downloader/response_status_count/200': 51,
 'elapsed_time_seconds': 67.603078,
 'feedexport/success_count/S3FeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 12, 17, 5, 34, 33, 371676, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 59868351,
 'httpcompression/response_count': 51,
 'log_count/DEBUG': 162,
 'log_count/INFO': 112,
 'memusage/max': 262877184,
 'memusage/startup': 124489728,
 'offsite/domains': 1,
 'offsite/filtered': 1,
 'request_depth_max': 2,
 'response_received_count': 51,
 'scheduler/dequeued': 51,
 'scheduler/dequeued/memory': 51,
 'scheduler/enqueued': 51,
 'scheduler/enqueued/memory': 51,
 'start_time': datetime.datetime(2025, 12, 17, 5, 33, 25, 768598, tzinfo=datetime.timezone.utc)}
2025-12-17 05:34:33 [scrapy.core.engine] INFO: Spider closed (finished)