2026-03-21 22:30:47 [scrapy.utils.log] INFO: Scrapy 2.11.1 started (bot: news_scraper)
2026-03-21 22:30:47 [scrapy.utils.log] INFO: Versions: lxml 6.0.2.0, libxml2 2.14.6, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 25.5.0, Python 3.11.13 (main, Aug 12 2025, 22:39:41) [GCC 14.2.0], pyOpenSSL 25.3.0 (OpenSSL 3.5.3 16 Sep 2025), cryptography 46.0.1, Platform Linux-5.15.0-164-generic-x86_64-with
2026-03-21 22:30:47 [scrapy.addons] INFO: Enabled addons:
[]
2026-03-21 22:30:47 [asyncio] DEBUG: Using selector: EpollSelector
2026-03-21 22:30:47 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2026-03-21 22:30:47 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2026-03-21 22:30:47 [scrapy.extensions.telnet] INFO: Telnet Password: 55df916df6877e80
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2026-03-21 22:30:47 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/endpoints.json
2026-03-21 22:30:47 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f282c95fec0>
2026-03-21 22:30:47 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json.gz
2026-03-21 22:30:47 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2026-03-21 22:30:47 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/partitions.json
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f282cab4a40>
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f282c894860>
2026-03-21 22:30:47 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f282cab47c0>
2026-03-21 22:30:47 [botocore.endpoint] DEBUG: Setting s3 timeout as (60, 60)
2026-03-21 22:30:47 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/_retry.json
2026-03-21 22:30:47 [botocore.client] DEBUG: Registering retry handlers for service: s3
2026-03-21 22:30:47 [botocore.utils] DEBUG: Registering S3 region redirector handler
2026-03-21 22:30:47 [botocore.utils] DEBUG: Registering S3Express Identity Resolver
2026-03-21 22:30:47 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle']
2026-03-21 22:30:47 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
 'BOT_NAME': 'news_scraper',
 'CLOSESPIDER_TIMEOUT': 1800,
 'CONCURRENT_REQUESTS': 4,
 'DOWNLOAD_DELAY': 2,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/opt/scrapyd/logs/news_scraper/yahooworld_timestamp/891016a4257511f1a8c68655d067ffdb.log',
 'NEWSPIDER_MODULE': 'news_scraper.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['news_scraper.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2026-03-21 22:30:48 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'news_scraper.middlewares.NewsScraperDownloaderMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-03-21 22:30:48 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2026-03-21 22:30:48 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2026-03-21 22:30:48 [scrapy.core.engine] INFO: Spider opened
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2026-03-21 22:30:48 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/endpoints.json
2026-03-21 22:30:48 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f282c95fec0>
2026-03-21 22:30:48 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json.gz
2026-03-21 22:30:48 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2026-03-21 22:30:48 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/partitions.json
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f282cab4a40>
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f282b8fade0>
2026-03-21 22:30:48 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f282cab47c0>
2026-03-21 22:30:48 [botocore.endpoint] DEBUG: Setting s3 timeout as (60, 60)
2026-03-21 22:30:48 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/_retry.json
2026-03-21 22:30:48 [botocore.client] DEBUG: Registering retry handlers for service: s3
2026-03-21 22:30:48 [botocore.utils] DEBUG: Registering S3 region redirector handler
2026-03-21 22:30:48 [botocore.utils] DEBUG: Registering S3Express Identity Resolver
2026-03-21 22:30:48 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-03-21 22:30:48 [yahooworld_timestamp] INFO: Spider opened: yahooworld_timestamp
2026-03-21 22:30:48 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6028
2026-03-21 22:30:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.yahoo.com/> (referer: None)
2026-03-21 22:30:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.yahoo.com/world/> (referer: https://news.yahoo.com/)
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/berlin-within-range-iranian-missiles-195913853.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/eid-without-toys-israeli-restrictions-180320265.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/kurds-celebrate-nowruz-amid-iran-162110603.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/studying-under-fire-israels-students-160000904.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/far-offences-germany-drop-slightly-152922834.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/sectors/energy/articles/united-mulls-strategy-fuel-costs-151430571.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://shopping.yahoo.com/home-garden/home-decor/articles/stucco-patio-perfect-style-those-150000247.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/sectors/energy/articles/diesel-prices-germany-surge-near-134422512.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/iran-warns-fresh-wave-attacks-134421068.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/finland-divers-learn-explore-icy-103312424.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://travel.yahoo.com/guides/canada/articles/canadas-coming-nature-destination-trending-103000915.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/germanys-pistorius-begins-asia-trip-101430214.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/israel-targets-sites-iran-lebanon-101430928.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/markets/commodities/articles/not-just-oil-3-critical-091501521.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/kindergarten-rishon-lezion-damaged-latest-090903702.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/crowd-near-old-city-dispersed-063422603.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://travel.yahoo.com/guides/articles/500-days-until-21st-century-021400033.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/live-possum-discovered-hiding-among-014848081.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/articles/max-watanabe-founded-max-factory-235515960.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/ukraine-hungary-oil-pipeline-row-201634671.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/australian-prime-minister-heckled-mosque-210429606.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/pakistan-flooding-kills-least-15-190920506.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/entertainment/celebrity/articles/chandler-powell-become-backbone-australia-190739292.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/japan-underground-flood-system-protects-183656558.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/sectors/technology/articles/sap-says-defense-becomes-fastest-182942266.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/cherry-trees-stolen-park-selfish-182048000.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/nato-withdraws-troops-iraq-due-180016123.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/poland-pulls-troops-iraq-over-180016897.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/friday-march-20-russia-war-173122990.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/listed-former-bank-become-ivy-171407527.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/economy/policy/articles/brazils-finance-minister-delay-divisive-170552809.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/spain-unveils-5bn-relief-plan-164434096.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/iran-khamenei-says-enemy-defeated-163853661.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/ukraine-says-red-team-forces-163441464.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/irans-khamenei-absent-persian-speech-161423944.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://travel.yahoo.com/guides/articles/ive-traveled-beyond-japans-popular-160132392.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://travel.yahoo.com/travel-deals/articles/could-win-free-trip-japan-154953235.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/hungarys-orb-n-accused-disloyalty-113459630.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/drifting-russian-tanker-entered-libya-153254889.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/diva-alert-rare-crab-traits-152228354.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/german-police-open-mouthed-discovery-144423147.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/texelis-scata-team-medium-heavy-143517443.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/france-takes-appropriate-measures-sailors-143235003.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://autos.yahoo.com/classic-and-collector/articles/america-never-got-tiny-diesel-143048409.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/russia-set-allow-military-free-142938016.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/economy/policy/articles/spain-unveils-5-billion-energy-142851565.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/israel-ambassador-russia-summoned-over-142615795.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/sectors/energy/articles/nigeria-faces-iran-war-oil-141626411.html
2026-03-21 22:30:53 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/israel-ranks-eighth-world-happiness-141554084.html
2026-03-21 22:30:54 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/germany-detains-hamas-arms-smuggling-141422549.html
2026-03-21 22:30:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://travel.yahoo.com/guides/articles/500-days-until-21st-century-021400033.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:30:54 [yahooworld_timestamp] INFO: 2026-03-21 is out of date range: from 2026-03-22 to 2026-03-22, skipping article: https://travel.yahoo.com/guides/articles/500-days-until-21st-century-021400033.html
2026-03-21 22:30:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/berlin-within-range-iranian-missiles-195913853.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:30:55 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/berlin-within-range-iranian-missiles-195913853.html
2026-03-21 22:31:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/live-possum-discovered-hiding-among-014848081.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:01 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/live-possum-discovered-hiding-among-014848081.html
2026-03-21 22:31:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/sectors/energy/articles/nigeria-faces-iran-war-oil-141626411.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:01 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/sectors/energy/articles/nigeria-faces-iran-war-oil-141626411.html
2026-03-21 22:31:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/crowd-near-old-city-dispersed-063422603.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:04 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/crowd-near-old-city-dispersed-063422603.html
2026-03-21 22:31:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/economy/policy/articles/spain-unveils-5-billion-energy-142851565.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:05 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/economy/policy/articles/spain-unveils-5-billion-energy-142851565.html
2026-03-21 22:31:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/germany-detains-hamas-arms-smuggling-141422549.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:07 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/germany-detains-hamas-arms-smuggling-141422549.html
2026-03-21 22:31:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://autos.yahoo.com/classic-and-collector/articles/america-never-got-tiny-diesel-143048409.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:07 [yahooworld_timestamp] INFO: 2026-03-20 is out of date range: from 2026-03-22 to 2026-03-22, skipping article: https://autos.yahoo.com/classic-and-collector/articles/america-never-got-tiny-diesel-143048409.html
2026-03-21 22:31:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/israel-ranks-eighth-world-happiness-141554084.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:10 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/israel-ranks-eighth-world-happiness-141554084.html
2026-03-21 22:31:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/israel-ambassador-russia-summoned-over-142615795.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:12 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/israel-ambassador-russia-summoned-over-142615795.html
2026-03-21 22:31:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/russia-set-allow-military-free-142938016.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:15 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/russia-set-allow-military-free-142938016.html
2026-03-21 22:31:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/france-takes-appropriate-measures-sailors-143235003.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:18 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/france-takes-appropriate-measures-sailors-143235003.html
2026-03-21 22:31:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/texelis-scata-team-medium-heavy-143517443.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:20 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/texelis-scata-team-medium-heavy-143517443.html
2026-03-21 22:31:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/german-police-open-mouthed-discovery-144423147.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:23 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/german-police-open-mouthed-discovery-144423147.html
2026-03-21 22:31:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://travel.yahoo.com/travel-deals/articles/could-win-free-trip-japan-154953235.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:24 [yahooworld_timestamp] INFO: 2026-03-20 is out of date range: from 2026-03-22 to 2026-03-22, skipping article: https://travel.yahoo.com/travel-deals/articles/could-win-free-trip-japan-154953235.html
2026-03-21 22:31:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/diva-alert-rare-crab-traits-152228354.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:25 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/diva-alert-rare-crab-traits-152228354.html
2026-03-21 22:31:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://travel.yahoo.com/guides/articles/ive-traveled-beyond-japans-popular-160132392.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:28 [yahooworld_timestamp] INFO: 2026-03-20 is out of date range: from 2026-03-22 to 2026-03-22, skipping article: https://travel.yahoo.com/guides/articles/ive-traveled-beyond-japans-popular-160132392.html
2026-03-21 22:31:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/drifting-russian-tanker-entered-libya-153254889.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:28 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/drifting-russian-tanker-entered-libya-153254889.html
2026-03-21 22:31:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/hungarys-orb-n-accused-disloyalty-113459630.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:30 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/hungarys-orb-n-accused-disloyalty-113459630.html
2026-03-21 22:31:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/irans-khamenei-absent-persian-speech-161423944.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:33 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/irans-khamenei-absent-persian-speech-161423944.html
2026-03-21 22:31:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/economy/policy/articles/brazils-finance-minister-delay-divisive-170552809.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:33 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/economy/policy/articles/brazils-finance-minister-delay-divisive-170552809.html
2026-03-21 22:31:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/ukraine-says-red-team-forces-163441464.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:35 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/ukraine-says-red-team-forces-163441464.html
2026-03-21 22:31:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/iran-khamenei-says-enemy-defeated-163853661.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:38 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/iran-khamenei-says-enemy-defeated-163853661.html
2026-03-21 22:31:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/spain-unveils-5bn-relief-plan-164434096.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:40 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/spain-unveils-5bn-relief-plan-164434096.html
2026-03-21 22:31:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/listed-former-bank-become-ivy-171407527.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:43 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/listed-former-bank-become-ivy-171407527.html
2026-03-21 22:31:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/friday-march-20-russia-war-173122990.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:46 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/friday-march-20-russia-war-173122990.html
2026-03-21 22:31:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/sectors/technology/articles/sap-says-defense-becomes-fastest-182942266.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:47 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/sectors/technology/articles/sap-says-defense-becomes-fastest-182942266.html
2026-03-21 22:31:48 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 29 pages/min), scraped 0 items (at 0 items/min)
2026-03-21 22:31:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/poland-pulls-troops-iraq-over-180016897.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:48 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/poland-pulls-troops-iraq-over-180016897.html
2026-03-21 22:31:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/nato-withdraws-troops-iraq-due-180016123.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:50 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/nato-withdraws-troops-iraq-due-180016123.html
2026-03-21 22:31:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/cherry-trees-stolen-park-selfish-182048000.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:53 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/cherry-trees-stolen-park-selfish-182048000.html
2026-03-21 22:31:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/japan-underground-flood-system-protects-183656558.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:56 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/japan-underground-flood-system-protects-183656558.html
2026-03-21 22:31:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/articles/chandler-powell-become-backbone-australia-190739292.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:31:59 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/articles/chandler-powell-become-backbone-australia-190739292.html
2026-03-21 22:32:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/pakistan-flooding-kills-least-15-190920506.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:01 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/pakistan-flooding-kills-least-15-190920506.html
2026-03-21 22:32:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/australian-prime-minister-heckled-mosque-210429606.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:03 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/australian-prime-minister-heckled-mosque-210429606.html
2026-03-21 22:32:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/markets/commodities/articles/not-just-oil-3-critical-091501521.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:04 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/markets/commodities/articles/not-just-oil-3-critical-091501521.html
2026-03-21 22:32:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/ukraine-hungary-oil-pipeline-row-201634671.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:06 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/ukraine-hungary-oil-pipeline-row-201634671.html
2026-03-21 22:32:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/entertainment/celebrity/articles/max-watanabe-founded-max-factory-235515960.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:09 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/entertainment/celebrity/articles/max-watanabe-founded-max-factory-235515960.html
2026-03-21 22:32:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://travel.yahoo.com/guides/canada/articles/canadas-coming-nature-destination-trending-103000915.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:10 [yahooworld_timestamp] INFO: 2026-03-21 is out of date range: from 2026-03-22 to 2026-03-22, skipping article: https://travel.yahoo.com/guides/canada/articles/canadas-coming-nature-destination-trending-103000915.html
2026-03-21 22:32:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/kindergarten-rishon-lezion-damaged-latest-090903702.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:11 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/kindergarten-rishon-lezion-damaged-latest-090903702.html
2026-03-21 22:32:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/israel-targets-sites-iran-lebanon-101430928.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:14 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/israel-targets-sites-iran-lebanon-101430928.html
2026-03-21 22:32:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/sectors/energy/articles/diesel-prices-germany-surge-near-134422512.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:14 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/sectors/energy/articles/diesel-prices-germany-surge-near-134422512.html
2026-03-21 22:32:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://shopping.yahoo.com/home-garden/home-decor/articles/stucco-patio-perfect-style-those-150000247.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:15 [yahooworld_timestamp] INFO: 2026-03-21 is out of date range: from 2026-03-22 to 2026-03-22, skipping article: https://shopping.yahoo.com/home-garden/home-decor/articles/stucco-patio-perfect-style-those-150000247.html
2026-03-21 22:32:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/sectors/energy/articles/united-mulls-strategy-fuel-costs-151430571.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:16 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/sectors/energy/articles/united-mulls-strategy-fuel-costs-151430571.html
2026-03-21 22:32:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/germanys-pistorius-begins-asia-trip-101430214.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:17 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/germanys-pistorius-begins-asia-trip-101430214.html
2026-03-21 22:32:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/finland-divers-learn-explore-icy-103312424.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:19 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/finland-divers-learn-explore-icy-103312424.html
2026-03-21 22:32:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/iran-warns-fresh-wave-attacks-134421068.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:22 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/iran-warns-fresh-wave-attacks-134421068.html
2026-03-21 22:32:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/far-offences-germany-drop-slightly-152922834.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:25 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/far-offences-germany-drop-slightly-152922834.html
2026-03-21 22:32:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/studying-under-fire-israels-students-160000904.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:27 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/studying-under-fire-israels-students-160000904.html
2026-03-21 22:32:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/kurds-celebrate-nowruz-amid-iran-162110603.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:30 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/kurds-celebrate-nowruz-amid-iran-162110603.html
2026-03-21 22:32:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/eid-without-toys-israeli-restrictions-180320265.html> (referer: https://news.yahoo.com/world/)
2026-03-21 22:32:32 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/eid-without-toys-israeli-restrictions-180320265.html
2026-03-21 22:32:32 [scrapy.core.engine] INFO: Closing spider (finished)
2026-03-21 22:32:32 [boto3.s3.transfer] DEBUG: Opting out of CRT Transfer Manager. Preferred client: auto, CRT available: False, Instance Optimized: False.
2026-03-21 22:32:32 [boto3.s3.transfer] DEBUG: Using default client. pid: 24995, thread: 139810505161528
2026-03-21 22:32:32 [s3transfer.utils] DEBUG: Acquiring 0
2026-03-21 22:32:32 [s3transfer.tasks] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f282bed62d0>}) about to wait for the following futures []
2026-03-21 22:32:32 [s3transfer.tasks] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f282bed62d0>}) done waiting for dependent futures
2026-03-21 22:32:32 [s3transfer.tasks] DEBUG: Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f282bed62d0>}) with kwargs {'client': <botocore.client.S3 object at 0x7f282c15abd0>, 'config': <boto3.s3.transfer.TransferConfig object at 0x7f282bec6390>, 'osutil': <s3transfer.utils.OSUtils object at 0x7f282bcf1190>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7f282c3572d0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f282bed62d0>}
2026-03-21 22:32:32 [s3transfer.futures] DEBUG: Submitting task PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f282c3572d0> for transfer request: 0.
2026-03-21 22:32:32 [s3transfer.utils] DEBUG: Acquiring 0
2026-03-21 22:32:32 [s3transfer.tasks] DEBUG: PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'extra_args': {}}) about to wait for the following futures []
2026-03-21 22:32:32 [s3transfer.utils] DEBUG: Releasing acquire 0/None
2026-03-21 22:32:32 [s3transfer.tasks] DEBUG: PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'extra_args': {}}) done waiting for dependent futures
2026-03-21 22:32:32 [s3transfer.tasks] DEBUG: Executing task PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7f282c15abd0>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x7f282b082ad0>, 'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'extra_args': {}}
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x7f282c972660>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x7f282c971940>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x7f282c973060>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x7f282c9718a0>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f282c973a60>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f282c4c3350>>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x7f282c4449d0>>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x7f282c9716c0>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f282c973c40>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f282c4c3350>>
2026-03-21 22:32:32 [botocore.regions] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'dagster-output-data', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://lake-api.actable.ai/', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2026-03-21 22:32:32 [botocore.regions] DEBUG: Endpoint provider result: https://lake-api.actable.ai/dagster-output-data
2026-03-21 22:32:32 [botocore.regions] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2026-03-21 22:32:32 [botocore.regions] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function conditionally_calculate_checksum at 0x7f282cdd8d60>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x7f282c971c60>
2026-03-21 22:32:32 [botocore.handlers] DEBUG: Adding expect 100 continue header to request.
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f282c4c3fd0>>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function add_recursion_detection_header at 0x7f282c95ff60>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x7f282c9731a0>
2026-03-21 22:32:32 [botocore.endpoint] DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.57 md/Botocore#1.34.162 ua/2.0 os/linux#5.15.0-164-generic md/arch#x86_64 lang/python#3.11.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.162', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7f282b082ad0>, 'auth_path': '/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'url': 'https://lake-api.actable.ai/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f282c1892d0>, 'has_streaming_input': True, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'dagster-output-data', 'params': {'Bucket': 'dagster-output-data', 'Key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl', 'Body': <s3transfer.utils.ReadFileChunk object at 0x7f282b082ad0>}}, 'input_params': {'Bucket': 'dagster-output-data', 'Key': 'yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl'}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function signal_not_transferring at 0x7f282bc840e0>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f282c189290>>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f282c4449d0>>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7f282c971580>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x7f282c973ba0>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event before-sign.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f282c4c3fd0>>
2026-03-21 22:32:32 [botocore.auth] DEBUG: Calculating signature using v4 auth.
2026-03-21 22:32:32 [botocore.auth] DEBUG: CanonicalRequest:
PUT
/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl

content-md5:1B2M2Y8AsgTpgAmY7PhCfg==
host:lake-api.actable.ai
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20260321T223232Z

content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2026-03-21 22:32:32 [botocore.auth] DEBUG: StringToSign:
AWS4-HMAC-SHA256
20260321T223232Z
20260321/us-east-1/s3/aws4_request
f7823b5865a9c7c273ca8d52d74c536176ae893b8e2962e568661bc58e036725
2026-03-21 22:32:32 [botocore.auth] DEBUG: Signature:
3be0e9db7412796b495d6ab5e79ec22dcba27417f82d60800a2301b854fd34bf
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function signal_transferring at 0x7f282bc84180>
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7f282c9739c0>
2026-03-21 22:32:32 [botocore.endpoint] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lake-api.actable.ai/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl, headers={'User-Agent': b'Boto3/1.34.57 md/Botocore#1.34.162 ua/2.0 os/linux#5.15.0-164-generic md/arch#x86_64 lang/python#3.11.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.162', 'Content-MD5': b'1B2M2Y8AsgTpgAmY7PhCfg==', 'Expect': b'100-continue', 'X-Amz-Date': b'20260321T223232Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=9zTeMQWwQiGhbKvMJ7uq/20260321/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=3be0e9db7412796b495d6ab5e79ec22dcba27417f82d60800a2301b854fd34bf', 'amz-sdk-invocation-id': b'6dfb9d71-4fa6-4362-bab5-70cc33bea827', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
2026-03-21 22:32:32 [botocore.httpsession] DEBUG: Certificate path: /usr/local/lib/python3.11/site-packages/certifi/cacert.pem
2026-03-21 22:32:32 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): lake-api.actable.ai:443
2026-03-21 22:32:32 [botocore.awsrequest] DEBUG: Waiting for 100 Continue response.
2026-03-21 22:32:32 [botocore.awsrequest] DEBUG: 100 Continue response seen, now sending request body.
2026-03-21 22:32:32 [urllib3.connectionpool] DEBUG: https://lake-api.actable.ai:443 "PUT /dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl HTTP/1.1" 200 0
2026-03-21 22:32:32 [botocore.parsers] DEBUG: Response headers: {'Server': 'nginx/1.18.0 (Ubuntu)', 'Date': 'Sat, 21 Mar 2026 22:32:32 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Accept-Ranges': 'bytes', 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Bucket-Region': 'us-east-1', 'X-Amz-Id-2': 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8', 'X-Amz-Request-Id': '189EFBF99F75A65A', 'X-Content-Type-Options': 'nosniff', 'X-Ratelimit-Limit': '3162', 'X-Ratelimit-Remaining': '3162', 'X-Xss-Protection': '1; mode=block'}
2026-03-21 22:32:32 [botocore.parsers] DEBUG: Response body:
b''
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f282c4c3d90>
2026-03-21 22:32:32 [botocore.retryhandler] DEBUG: No retry needed.
2026-03-21 22:32:32 [botocore.hooks] DEBUG: Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f282c4c3350>>
2026-03-21 22:32:32 [s3transfer.utils] DEBUG: Releasing acquire 0/None
2026-03-21 22:32:32 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (0 items) in: s3://dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_891016a4257511f1a8c68655d067ffdb_scheduled_2026-03-22.jl
2026-03-21 22:32:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 44300,
 'downloader/request_count': 52,
 'downloader/request_method_count/GET': 52,
 'downloader/response_bytes': 8013995,
 'downloader/response_count': 52,
 'downloader/response_status_count/200': 52,
 'elapsed_time_seconds': 104.591725,
 'feedexport/success_count/S3FeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2026, 3, 21, 22, 32, 32, 626185, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 58348166,
 'httpcompression/response_count': 52,
 'log_count/DEBUG': 162,
 'log_count/INFO': 113,
 'memusage/max': 236359680,
 'memusage/startup': 124751872,
 'request_depth_max': 2,
 'response_received_count': 52,
 'scheduler/dequeued': 52,
 'scheduler/dequeued/memory': 52,
 'scheduler/enqueued': 52,
 'scheduler/enqueued/memory': 52,
 'start_time': datetime.datetime(2026, 3, 21, 22, 30, 48, 34460, tzinfo=datetime.timezone.utc)}
2026-03-21 22:32:32 [scrapy.core.engine] INFO: Spider closed (finished)