2025-09-25 20:16:50 [scrapy.utils.log] INFO: Scrapy 2.11.1 started (bot: news_scraper)
2025-09-25 20:16:50 [scrapy.utils.log] INFO: Versions: lxml 6.0.0.0, libxml2 2.14.4, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 25.5.0, Python 3.11.13 (main, Jul 15 2025, 19:29:01) [GCC 14.2.0], pyOpenSSL 25.1.0 (OpenSSL 3.5.1 1 Jul 2025), cryptography 45.0.5, Platform Linux-5.15.0-139-generic-x86_64-with
2025-09-25 20:16:50 [scrapy.addons] INFO: Enabled addons:
[]
2025-09-25 20:16:50 [asyncio] DEBUG: Using selector: EpollSelector
2025-09-25 20:16:50 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2025-09-25 20:16:50 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2025-09-25 20:16:50 [scrapy.extensions.telnet] INFO: Telnet Password: 3bb79e65287b5d5e
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2025-09-25 20:16:50 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/endpoints.json
2025-09-25 20:16:50 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f7036677420>
2025-09-25 20:16:50 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json.gz
2025-09-25 20:16:50 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2025-09-25 20:16:50 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/partitions.json
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f70367d7f60>
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f70365b3d80>
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f70367d7ce0>
2025-09-25 20:16:50 [botocore.endpoint] DEBUG: Setting s3 timeout as (60, 60)
2025-09-25 20:16:50 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/_retry.json
2025-09-25 20:16:50 [botocore.client] DEBUG: Registering retry handlers for service: s3
2025-09-25 20:16:50 [botocore.utils] DEBUG: Registering S3 region redirector handler
2025-09-25 20:16:50 [botocore.utils] DEBUG: Registering S3Express Identity Resolver
2025-09-25 20:16:50 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle']
2025-09-25 20:16:50 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
 'BOT_NAME': 'news_scraper',
 'CLOSESPIDER_TIMEOUT': 1800,
 'CONCURRENT_REQUESTS': 4,
 'DOWNLOAD_DELAY': 2,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/opt/scrapyd/logs/news_scraper/yahooworld_timestamp/7d759e229a4c11f086971e907748958e.log',
 'NEWSPIDER_MODULE': 'news_scraper.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['news_scraper.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2025-09-25 20:16:50 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'news_scraper.middlewares.NewsScraperDownloaderMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2025-09-25 20:16:50 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-09-25 20:16:50 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2025-09-25 20:16:50 [scrapy.core.engine] INFO: Spider opened
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2025-09-25 20:16:50 [botocore.hooks] DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2025-09-25 20:16:50 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/endpoints.json
2025-09-25 20:16:51 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2025-09-25 20:16:51 [botocore.hooks] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f7036677420>
2025-09-25 20:16:51 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json.gz
2025-09-25 20:16:51 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2025-09-25 20:16:51 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/partitions.json
2025-09-25 20:16:51 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f70367d7f60>
2025-09-25 20:16:51 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f70357ae340>
2025-09-25 20:16:51 [botocore.hooks] DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f70367d7ce0>
2025-09-25 20:16:51 [botocore.endpoint] DEBUG: Setting s3 timeout as (60, 60)
2025-09-25 20:16:51 [botocore.loaders] DEBUG: Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/_retry.json
2025-09-25 20:16:51 [botocore.client] DEBUG: Registering retry handlers for service: s3
2025-09-25 20:16:51 [botocore.utils] DEBUG: Registering S3 region redirector handler
2025-09-25 20:16:51 [botocore.utils] DEBUG: Registering S3Express Identity Resolver
2025-09-25 20:16:51 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-09-25 20:16:51 [yahooworld_timestamp] INFO: Spider opened: yahooworld_timestamp
2025-09-25 20:16:51 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6032
2025-09-25 20:16:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.yahoo.com/> (referer: None)
2025-09-25 20:16:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.yahoo.com/world/> (referer: https://news.yahoo.com/)
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/eight-injured-explosion-destroys-building-195920131.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/expert-report-alleges-systematic-abuse-195920403.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://autos.yahoo.com/people-and-culture/articles/lambo-guy-adrian-portelli-buys-183000517.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/israeli-strike-yemens-houthis-reportedly-194718767.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/byd-overtakes-tesla-europe-ev-193107388.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/vote-determine-eurovision-2026-contestants-191435068.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/dozens-palestinians-reported-killed-israeli-191435292.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/article/achraf-hakimi-publicly-addresses-sexual-191400862.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/madagascar-imposes-nighttime-curfew-violent-190312833.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/shocking-scenes-destruction-typhoon-ragasa-132233851.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/irans-vice-president-says-wont-183858693.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/majority-germans-think-israel-committing-183339691.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/three-judges-elected-germanys-top-182934404.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/mali-ends-counterterror-cooperation-france-180145741.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/much-does-iran-rebuilding-ballistic-180143731.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/russia-extends-fuel-export-ban-175932903.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/doug-ford-plans-ban-speed-174537061.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/moldova-ruling-pro-eu-party-173443595.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/german-lawmakers-elect-top-court-172939494.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/amid-drone-attacks-activists-confront-171421724.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/sea-life-thriving-unexploded-nazi-170607428.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/hive-hit-3-global-bitcoin-170145762.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/germanys-spd-ups-pressure-merz-165936378.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/drones-disrupted-danish-airports-turned-165439917.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/nato-secretary-general-takes-danish-161418815.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/un-court-hears-case-alleged-160609016.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/netanyahu-banned-travel-slovenia-ljubljana-155256001.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/slovenia-declares-netanyahu-persona-non-154428695.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://autos.yahoo.com/classic-and-collector/articles/zagato-debut-limited-run-gullwing-153000549.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/french-navy-seizes-nearly-10-113218569.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/germanys-merz-backs-using-frozen-151440810.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/israeli-airstrikes-target-yemens-houthi-145928145.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/slovenia-bans-israeli-prime-minister-145032404.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/jordans-curriculum-full-antisemitic-anti-144044640.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/nicolas-sarkozy-found-guilty-criminal-085514717.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/planned-force-israel-end-gaza-135827696.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/germany-spend-35-billion-space-132637646.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/lifestyle/articles/trade-lake-como-crowds-hidden-132125868.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/germany-rearm-drone-defence-light-122631069.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/man-arrested-threatening-kill-netanyahu-121043655.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/iaf-hits-170-hamas-targets-120332599.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://autos.yahoo.com/policy-and-environment/articles/stellantis-idle-european-plants-120000810.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/lifestyle/articles/next-generation-cruise-ships-powered-120000417.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/weird-glass-australia-appears-giant-115944987.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/vietnam-jails-43-3-8b-114715634.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/seychelles-holds-early-voting-many-112623286.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://sports.yahoo.com/article/2025-morocco-eases-entry-fans-112500711.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/south-korea-crypto-regulation-limbo-105317106.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://www.yahoo.com/news/articles/flights-italy-face-disruption-nationwide-055742191.html
2025-09-25 20:16:58 [yahooworld_timestamp] INFO: URL: https://finance.yahoo.com/news/germany-must-move-quicker-reforms-100938895.html
2025-09-25 20:17:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/eight-injured-explosion-destroys-building-195920131.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:00 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=Ubk9VBI&done=https%3A%2F%2Ffinance.yahoo.com%2Fnews%2Fgermany-must-move-quicker-reforms-100938895.html> from <GET https://finance.yahoo.com/news/germany-must-move-quicker-reforms-100938895.html>
2025-09-25 20:17:00 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/eight-injured-explosion-destroys-building-195920131.html
2025-09-25 20:17:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/three-judges-elected-germanys-top-182934404.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:03 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/three-judges-elected-germanys-top-182934404.html
2025-09-25 20:17:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/majority-germans-think-israel-committing-183339691.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:07 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/majority-germans-think-israel-committing-183339691.html
2025-09-25 20:17:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://finance.yahoo.com/news/germany-must-move-quicker-reforms-100938895.html?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnlhaG9vLmNvbS93b3JsZC8&guce_referrer_sig=AQAAAC5oLctG_gQVG56q3T6Qg9_OiPXj9QMkoVXEwL9NLG6aK08WSdBj894Cf5_66ogPxTiD-Vskt_ts-zxtpxNGEn81eyDKwf8lTtfzICGYVD7-gxHYt8HFT9IhlneU8I8W4bIiSWhGCxOsK936hQxF1JifHKL9F15cN5jX83T0Z_WJ> from <GET https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=Ubk9VBI&done=https%3A%2F%2Ffinance.yahoo.com%2Fnews%2Fgermany-must-move-quicker-reforms-100938895.html>
2025-09-25 20:17:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/irans-vice-president-says-wont-183858693.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:08 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/irans-vice-president-says-wont-183858693.html
2025-09-25 20:17:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/article/2025-morocco-eases-entry-fans-112500711.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:10 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/article/2025-morocco-eases-entry-fans-112500711.html
2025-09-25 20:17:12 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=Qtyr_CY&done=https%3A%2F%2Ffinance.yahoo.com%2Fnews%2Fsouth-korea-crypto-regulation-limbo-105317106.html> from <GET https://finance.yahoo.com/news/south-korea-crypto-regulation-limbo-105317106.html>
2025-09-25 20:17:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/flights-italy-face-disruption-nationwide-055742191.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:12 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/flights-italy-face-disruption-nationwide-055742191.html
2025-09-25 20:17:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/seychelles-holds-early-voting-many-112623286.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:14 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/seychelles-holds-early-voting-many-112623286.html
2025-09-25 20:17:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/weird-glass-australia-appears-giant-115944987.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:17 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/weird-glass-australia-appears-giant-115944987.html
2025-09-25 20:17:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/germany-must-move-quicker-reforms-100938895.html?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnlhaG9vLmNvbS93b3JsZC8&guce_referrer_sig=AQAAAC5oLctG_gQVG56q3T6Qg9_OiPXj9QMkoVXEwL9NLG6aK08WSdBj894Cf5_66ogPxTiD-Vskt_ts-zxtpxNGEn81eyDKwf8lTtfzICGYVD7-gxHYt8HFT9IhlneU8I8W4bIiSWhGCxOsK936hQxF1JifHKL9F15cN5jX83T0Z_WJ> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:19 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/germany-must-move-quicker-reforms-100938895.html?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnlhaG9vLmNvbS93b3JsZC8&guce_referrer_sig=AQAAAC5oLctG_gQVG56q3T6Qg9_OiPXj9QMkoVXEwL9NLG6aK08WSdBj894Cf5_66ogPxTiD-Vskt_ts-zxtpxNGEn81eyDKwf8lTtfzICGYVD7-gxHYt8HFT9IhlneU8I8W4bIiSWhGCxOsK936hQxF1JifHKL9F15cN5jX83T0Z_WJ
2025-09-25 20:17:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/lifestyle/articles/next-generation-cruise-ships-powered-120000417.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:20 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/lifestyle/articles/next-generation-cruise-ships-powered-120000417.html
2025-09-25 20:17:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://autos.yahoo.com/policy-and-environment/articles/stellantis-idle-european-plants-120000810.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:20 [yahooworld_timestamp] INFO: 2025-09-25 is out of date range: from 2025-09-26 to 2025-09-26, skipping article: https://autos.yahoo.com/policy-and-environment/articles/stellantis-idle-european-plants-120000810.html
2025-09-25 20:17:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/iaf-hits-170-hamas-targets-120332599.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:23 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/iaf-hits-170-hamas-targets-120332599.html
2025-09-25 20:17:24 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://finance.yahoo.com/news/south-korea-crypto-regulation-limbo-105317106.html?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnlhaG9vLmNvbS93b3JsZC8&guce_referrer_sig=AQAAAC5oLctG_gQVG56q3T6Qg9_OiPXj9QMkoVXEwL9NLG6aK08WSdBj894Cf5_66ogPxTiD-Vskt_ts-zxtpxNGEn81eyDKwf8lTtfzICGYVD7-gxHYt8HFT9IhlneU8I8W4bIiSWhGCxOsK936hQxF1JifHKL9F15cN5jX83T0Z_WJ> from <GET https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=Qtyr_CY&done=https%3A%2F%2Ffinance.yahoo.com%2Fnews%2Fsouth-korea-crypto-regulation-limbo-105317106.html>
2025-09-25 20:17:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/man-arrested-threatening-kill-netanyahu-121043655.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:25 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/man-arrested-threatening-kill-netanyahu-121043655.html
2025-09-25 20:17:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/germany-rearm-drone-defence-light-122631069.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:28 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/germany-rearm-drone-defence-light-122631069.html
2025-09-25 20:17:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/vietnam-jails-43-3-8b-114715634.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:29 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/vietnam-jails-43-3-8b-114715634.html
2025-09-25 20:17:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/lifestyle/articles/trade-lake-como-crowds-hidden-132125868.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:30 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/lifestyle/articles/trade-lake-como-crowds-hidden-132125868.html
2025-09-25 20:17:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/south-korea-crypto-regulation-limbo-105317106.html?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnlhaG9vLmNvbS93b3JsZC8&guce_referrer_sig=AQAAAC5oLctG_gQVG56q3T6Qg9_OiPXj9QMkoVXEwL9NLG6aK08WSdBj894Cf5_66ogPxTiD-Vskt_ts-zxtpxNGEn81eyDKwf8lTtfzICGYVD7-gxHYt8HFT9IhlneU8I8W4bIiSWhGCxOsK936hQxF1JifHKL9F15cN5jX83T0Z_WJ> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:31 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/south-korea-crypto-regulation-limbo-105317106.html?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLnlhaG9vLmNvbS93b3JsZC8&guce_referrer_sig=AQAAAC5oLctG_gQVG56q3T6Qg9_OiPXj9QMkoVXEwL9NLG6aK08WSdBj894Cf5_66ogPxTiD-Vskt_ts-zxtpxNGEn81eyDKwf8lTtfzICGYVD7-gxHYt8HFT9IhlneU8I8W4bIiSWhGCxOsK936hQxF1JifHKL9F15cN5jX83T0Z_WJ
2025-09-25 20:17:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/germany-spend-35-billion-space-132637646.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:33 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/germany-spend-35-billion-space-132637646.html
2025-09-25 20:17:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/planned-force-israel-end-gaza-135827696.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:36 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/planned-force-israel-end-gaza-135827696.html
2025-09-25 20:17:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/nicolas-sarkozy-found-guilty-criminal-085514717.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:38 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/nicolas-sarkozy-found-guilty-criminal-085514717.html
2025-09-25 20:17:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/jordans-curriculum-full-antisemitic-anti-144044640.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:41 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/jordans-curriculum-full-antisemitic-anti-144044640.html
2025-09-25 20:17:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/slovenia-bans-israeli-prime-minister-145032404.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:42 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/slovenia-bans-israeli-prime-minister-145032404.html
2025-09-25 20:17:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://autos.yahoo.com/classic-and-collector/articles/zagato-debut-limited-run-gullwing-153000549.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:44 [yahooworld_timestamp] INFO: 2025-09-25 is out of date range: from 2025-09-26 to 2025-09-26, skipping article: https://autos.yahoo.com/classic-and-collector/articles/zagato-debut-limited-run-gullwing-153000549.html
2025-09-25 20:17:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/israeli-airstrikes-target-yemens-houthi-145928145.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:45 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/israeli-airstrikes-target-yemens-houthi-145928145.html
2025-09-25 20:17:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/germanys-merz-backs-using-frozen-151440810.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:47 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/germanys-merz-backs-using-frozen-151440810.html
2025-09-25 20:17:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/french-navy-seizes-nearly-10-113218569.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:49 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/french-navy-seizes-nearly-10-113218569.html
2025-09-25 20:17:51 [scrapy.extensions.logstats] INFO: Crawled 28 pages (at 28 pages/min), scraped 0 items (at 0 items/min)
2025-09-25 20:17:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/slovenia-declares-netanyahu-persona-non-154428695.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:52 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/slovenia-declares-netanyahu-persona-non-154428695.html
2025-09-25 20:17:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/netanyahu-banned-travel-slovenia-ljubljana-155256001.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:55 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/netanyahu-banned-travel-slovenia-ljubljana-155256001.html
2025-09-25 20:17:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/un-court-hears-case-alleged-160609016.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:57 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/un-court-hears-case-alleged-160609016.html
2025-09-25 20:17:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://finance.yahoo.com/news/hive-hit-3-global-bitcoin-170145762.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:58 [yahooworld_timestamp] INFO: Invalid article: https://finance.yahoo.com/news/hive-hit-3-global-bitcoin-170145762.html
2025-09-25 20:17:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/nato-secretary-general-takes-danish-161418815.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:17:59 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/nato-secretary-general-takes-danish-161418815.html
2025-09-25 20:18:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/drones-disrupted-danish-airports-turned-165439917.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:02 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/drones-disrupted-danish-airports-turned-165439917.html
2025-09-25 20:18:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/germanys-spd-ups-pressure-merz-165936378.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:05 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/germanys-spd-ups-pressure-merz-165936378.html
2025-09-25 20:18:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/sea-life-thriving-unexploded-nazi-170607428.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:08 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/sea-life-thriving-unexploded-nazi-170607428.html
2025-09-25 20:18:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/amid-drone-attacks-activists-confront-171421724.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:10 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/amid-drone-attacks-activists-confront-171421724.html
2025-09-25 20:18:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/german-lawmakers-elect-top-court-172939494.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:12 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/german-lawmakers-elect-top-court-172939494.html
2025-09-25 20:18:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/moldova-ruling-pro-eu-party-173443595.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:15 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/moldova-ruling-pro-eu-party-173443595.html
2025-09-25 20:18:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/doug-ford-plans-ban-speed-174537061.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:18 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/doug-ford-plans-ban-speed-174537061.html
2025-09-25 20:18:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/russia-extends-fuel-export-ban-175932903.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:20 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/russia-extends-fuel-export-ban-175932903.html
2025-09-25 20:18:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/much-does-iran-rebuilding-ballistic-180143731.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:23 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/much-does-iran-rebuilding-ballistic-180143731.html
2025-09-25 20:18:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sports.yahoo.com/article/achraf-hakimi-publicly-addresses-sexual-191400862.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:23 [yahooworld_timestamp] INFO: Invalid article: https://sports.yahoo.com/article/achraf-hakimi-publicly-addresses-sexual-191400862.html
2025-09-25 20:18:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/mali-ends-counterterror-cooperation-france-180145741.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:25 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/mali-ends-counterterror-cooperation-france-180145741.html
2025-09-25 20:18:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/shocking-scenes-destruction-typhoon-ragasa-132233851.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:28 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/shocking-scenes-destruction-typhoon-ragasa-132233851.html
2025-09-25 20:18:28 [scrapy.core.scraper] ERROR: Error downloading <GET https://finance.yahoo.com/news/byd-overtakes-tesla-europe-ev-193107388.html>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1857, in _inlineCallbacks
    result = context.run(gen.send, result)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_response
    method(request=request, response=response, spider=spider)
  File "/usr/local/lib/python3.11/site-packages/scrapy/downloadermiddlewares/httpcompression.py", line 90, in process_response
    decoded_body = self._decode(
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/scrapy/downloadermiddlewares/httpcompression.py", line 130, in _decode
    return gunzip(body, max_size=max_size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/scrapy/utils/gz.py", line 21, in gunzip
    chunk = f.read1(_CHUNK_SIZE)
            ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/gzip.py", line 314, in read1
    return self._buffer.read1(size)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/gzip.py", line 499, in read
    if not self._read_gzip_header():
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/gzip.py", line 468, in _read_gzip_header
    last_mtime = _read_gzip_header(self._fp)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/gzip.py", line 428, in _read_gzip_header
    raise BadGzipFile('Not a gzipped file (%r)' % magic)
gzip.BadGzipFile: Not a gzipped file (b'<h')
2025-09-25 20:18:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/madagascar-imposes-nighttime-curfew-violent-190312833.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:31 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/madagascar-imposes-nighttime-curfew-violent-190312833.html
2025-09-25 20:18:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://autos.yahoo.com/people-and-culture/articles/lambo-guy-adrian-portelli-buys-183000517.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:33 [yahooworld_timestamp] INFO: 2025-09-26 02:57:40 smaller than 2025-09-26 03:01:00
2025-09-25 20:18:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/dozens-palestinians-reported-killed-israeli-191435292.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:34 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/dozens-palestinians-reported-killed-israeli-191435292.html
2025-09-25 20:18:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/vote-determine-eurovision-2026-contestants-191435068.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:36 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/vote-determine-eurovision-2026-contestants-191435068.html
2025-09-25 20:18:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/israeli-strike-yemens-houthis-reportedly-194718767.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:38 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/israeli-strike-yemens-houthis-reportedly-194718767.html
2025-09-25 20:18:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.yahoo.com/news/articles/expert-report-alleges-systematic-abuse-195920403.html> (referer: https://news.yahoo.com/world/)
2025-09-25 20:18:40 [yahooworld_timestamp] INFO: Invalid article: https://www.yahoo.com/news/articles/expert-report-alleges-systematic-abuse-195920403.html
2025-09-25 20:18:40 [scrapy.core.engine] INFO: Closing spider (finished)
2025-09-25 20:18:40 [boto3.s3.transfer] DEBUG: Opting out of CRT Transfer Manager. Preferred client: auto, CRT available: False, Instance Optimized: False.
2025-09-25 20:18:40 [boto3.s3.transfer] DEBUG: Using default client. pid: 171336, thread: 140119907916600
2025-09-25 20:18:40 [s3transfer.utils] DEBUG: Acquiring 0
2025-09-25 20:18:40 [s3transfer.tasks] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f7034dc0490>}) about to wait for the following futures []
2025-09-25 20:18:40 [s3transfer.tasks] DEBUG: UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f7034dc0490>}) done waiting for dependent futures
2025-09-25 20:18:40 [s3transfer.tasks] DEBUG: Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f7034dc0490>}) with kwargs {'client': <botocore.client.S3 object at 0x7f7036457990>, 'config': <boto3.s3.transfer.TransferConfig object at 0x7f7034dc0a50>, 'osutil': <s3transfer.utils.OSUtils object at 0x7f70363fe810>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7f7034dc2c10>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f7034dc0490>}
2025-09-25 20:18:40 [s3transfer.futures] DEBUG: Submitting task PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f7034dc2c10> for transfer request: 0.
2025-09-25 20:18:40 [s3transfer.utils] DEBUG: Acquiring 0
2025-09-25 20:18:40 [s3transfer.tasks] DEBUG: PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'extra_args': {}}) about to wait for the following futures []
2025-09-25 20:18:40 [s3transfer.utils] DEBUG: Releasing acquire 0/None
2025-09-25 20:18:40 [s3transfer.tasks] DEBUG: PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'extra_args': {}}) done waiting for dependent futures
2025-09-25 20:18:40 [s3transfer.tasks] DEBUG: Executing task PutObjectTask(transfer_id=0, {'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7f7036457990>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x7f70363a4450>, 'bucket': 'dagster-output-data', 'key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'extra_args': {}}
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x7f703669dbc0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x7f703669cea0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x7f703669e5c0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x7f703669ce00>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f703669efc0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f70363d1110>>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x7f70361a90d0>>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x7f703669cc20>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f703669f1a0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f70363d1110>>
2025-09-25 20:18:40 [botocore.regions] DEBUG: Calling endpoint provider with parameters: {'Bucket': 'dagster-output-data', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://lake-api.actable.ai/', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2025-09-25 20:18:40 [botocore.regions] DEBUG: Endpoint provider result: https://lake-api.actable.ai/dagster-output-data
2025-09-25 20:18:40 [botocore.regions] DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2025-09-25 20:18:40 [botocore.regions] DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function conditionally_calculate_checksum at 0x7f7036b042c0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x7f703669d1c0>
2025-09-25 20:18:40 [botocore.handlers] DEBUG: Adding expect 100 continue header to request.
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f70363cd1d0>>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function add_recursion_detection_header at 0x7f70366774c0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x7f703669e700>
2025-09-25 20:18:40 [botocore.endpoint] DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.57 md/Botocore#1.34.162 ua/2.0 os/linux#5.15.0-139-generic md/arch#x86_64 lang/python#3.11.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.162', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7f70363a4450>, 'auth_path': '/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'url': 'https://lake-api.actable.ai/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f70363fe050>, 'has_streaming_input': True, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'dagster-output-data', 'params': {'Bucket': 'dagster-output-data', 'Key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl', 'Body': <s3transfer.utils.ReadFileChunk object at 0x7f70363a4450>}}, 'input_params': {'Bucket': 'dagster-output-data', 'Key': 'yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl'}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function signal_not_transferring at 0x7f70359b7600>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f70363fe010>>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f70361a90d0>>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7f703669cae0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x7f703669f100>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event before-sign.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f70363cd1d0>>
2025-09-25 20:18:40 [botocore.auth] DEBUG: Calculating signature using v4 auth.
2025-09-25 20:18:40 [botocore.auth] DEBUG: CanonicalRequest:
PUT
/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl

content-md5:1B2M2Y8AsgTpgAmY7PhCfg==
host:lake-api.actable.ai
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20250925T201840Z

content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2025-09-25 20:18:40 [botocore.auth] DEBUG: StringToSign:
AWS4-HMAC-SHA256
20250925T201840Z
20250925/us-east-1/s3/aws4_request
cebef31ffab22361568e7ce07794c23686496312757a33db330e0a926960dab5
2025-09-25 20:18:40 [botocore.auth] DEBUG: Signature:
9aee30a9a7cfb65707579b3326a6a30b485484fcec61408ed8380ec7a0475189
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function signal_transferring at 0x7f70359b76a0>
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7f703669ef20>
2025-09-25 20:18:40 [botocore.endpoint] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lake-api.actable.ai/dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl, headers={'User-Agent': b'Boto3/1.34.57 md/Botocore#1.34.162 ua/2.0 os/linux#5.15.0-139-generic md/arch#x86_64 lang/python#3.11.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.162', 'Content-MD5': b'1B2M2Y8AsgTpgAmY7PhCfg==', 'Expect': b'100-continue', 'X-Amz-Date': b'20250925T201840Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=9zTeMQWwQiGhbKvMJ7uq/20250925/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=9aee30a9a7cfb65707579b3326a6a30b485484fcec61408ed8380ec7a0475189', 'amz-sdk-invocation-id': b'33d427e6-4c0c-434b-942c-e312523e4034', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
2025-09-25 20:18:40 [botocore.httpsession] DEBUG: Certificate path: /usr/local/lib/python3.11/site-packages/certifi/cacert.pem
2025-09-25 20:18:40 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): lake-api.actable.ai:443
2025-09-25 20:18:40 [botocore.awsrequest] DEBUG: Waiting for 100 Continue response.
2025-09-25 20:18:40 [botocore.awsrequest] DEBUG: 100 Continue response seen, now sending request body.
2025-09-25 20:18:40 [urllib3.connectionpool] DEBUG: https://lake-api.actable.ai:443 "PUT /dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl HTTP/1.1" 200 0
2025-09-25 20:18:40 [botocore.parsers] DEBUG: Response headers: {'Server': 'nginx/1.24.0 (Ubuntu)', 'Date': 'Thu, 25 Sep 2025 20:18:40 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Accept-Ranges': 'bytes', 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Vary': 'Origin, Accept-Encoding', 'X-Amz-Bucket-Region': 'us-east-1', 'X-Amz-Id-2': 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8', 'X-Amz-Request-Id': '18689FF31143D9BC', 'X-Content-Type-Options': 'nosniff', 'X-Ratelimit-Limit': '25637', 'X-Ratelimit-Remaining': '25637', 'X-Xss-Protection': '1; mode=block'}
2025-09-25 20:18:40 [botocore.parsers] DEBUG: Response body:
b''
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f70363cd090>
2025-09-25 20:18:40 [botocore.retryhandler] DEBUG: No retry needed.
2025-09-25 20:18:40 [botocore.hooks] DEBUG: Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f70363d1110>>
2025-09-25 20:18:40 [s3transfer.utils] DEBUG: Releasing acquire 0/None
2025-09-25 20:18:40 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (0 items) in: s3://dagster-output-data/yahooworld_timestamp/yahooworld_timestamp_7d759e229a4c11f086971e907748958e_scheduled_2025-09-26.jl
2025-09-25 20:18:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 62992,
 'downloader/request_count': 56,
 'downloader/request_method_count/GET': 56,
 'downloader/response_bytes': 7395114,
 'downloader/response_count': 56,
 'downloader/response_status_count/200': 51,
 'downloader/response_status_count/302': 2,
 'downloader/response_status_count/307': 2,
 'downloader/response_status_count/404': 1,
 'elapsed_time_seconds': 109.807716,
 'feedexport/success_count/S3FeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 9, 25, 20, 18, 40, 780778, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 50581185,
 'httpcompression/response_count': 51,
 'log_count/DEBUG': 165,
 'log_count/ERROR': 1,
 'log_count/INFO': 112,
 'memusage/max': 220266496,
 'memusage/startup': 124133376,
 'request_depth_max': 2,
 'response_received_count': 51,
 'scheduler/dequeued': 56,
 'scheduler/dequeued/memory': 56,
 'scheduler/enqueued': 56,
 'scheduler/enqueued/memory': 56,
 'start_time': datetime.datetime(2025, 9, 25, 20, 16, 50, 973062, tzinfo=datetime.timezone.utc)}
2025-09-25 20:18:40 [scrapy.core.engine] INFO: Spider closed (finished)