Ruby on Rails App and API for fetching content from social networks and converting it to RSS.
Overview:
Display a list of content from a social network profile or search page as an RSS feed.
System Requirements:
Ruby on Rails 6.1.
Hosted on Heroku, using Postgres.
Background workers using Sidekiq, Sidekiq Scheduler and Redis.
App Requirements:
User authentication using Devise.
User must confirm email before access is allowed.
Generate an API key per user.
Connect to Social networks using oAuth, related to a single user account.
Support for Facebook, Twitter, Instagram, YouTube, LinkedIn, Reddit.
Notify user via email when a source cannot be fetched (there is a problem).
Notify user via email when their social network credentials need refreshing.
API Requirements
Allow a POST request with a URL and API key.
Fetch the URL content using the oAuth credentials to connect to the social network’s API or via a logged in HTTP request.
Configurable rules for parsing content per social network and per page type (see below for list of examples)
Parse the content into a list of items (using Nokogiri and xpath for HTTP) and each item having:
Title
Description
Author
Image
URL
Media (Video or Audio URL)
Cache Save: Save the requested URL and content for future requests.
Cache Use: Future requests should use the last saved content rather than directly requesting to the social network page.
Cache Update: Update the requested URL and content every 15 minutes (configurable time interval in the app)
Cache Expiry: Remove the requested URL and content after 3 days (configurable time interval in the app) of no new requests for that URL.
Investigate using a 3rd party such as https://www.scraperapi.com/ or https://www.scrapingbee.com/ for using proxies and/or anti-bot mechanisms to collect the data without being blocked.
