Downloads and statistics API #458

Activity

1. dcoutts commented on May 15, 2016

   

   dcoutts

   on May 15, 2016

   Contributor

   More actions

   Yes I'd very much like to expose download stats in a convenient form. We collect quite a bit of detail in the download stats feature but not a lot is exposed yet.

   @yamadapc you're most welcome to have a go at this. Ask in #hackage if you want advice.
2. yamadapc commented on Aug 28, 2016

   

   yamadapc

   on Aug 28, 2016

   ContributorAuthor

   More actions

   Updating this.

   Since there seems to be a fastly CDN in front of Hackage now, exposing the stats through the API isn't worth the trouble (since they'll be incorrect).

   Instead, it gets closer to what NPM seems to do by generating the download stats data from the CDN's logs.

   It seems there're ways to enable log aggregation from fastly as outlined in:

   • https://docs.fastly.com/guides/streaming-logs/custom-log-formats
   • https://docs.fastly.com/api/logging

   Another system, like NPM's download-counts would then run every scheduled time, parse the logs and generate statistics for the download counts. Depending on how the logs are structured, I'd guess, there could even be some re-use of NPM's existing tooling.
3. chrissound commented on May 20, 2017

   

   chrissound

   on May 20, 2017

   More actions

   I would be quite interested in this. It'd would be great to compare packages with this data. I've even got a web frontend up for a similar project (open source):
   Example: https://trycatchchris.co.uk/archpackagecompare/comparePackage/gnome-terminal/lxterminal/rxvt/rxvt-unicode/st/terminator/termite/xterm
   Code: https://github.com/chrissound/ArchPackageCompareStats

   I'm not too familiar with all these logging services though - are these all paid services?
4. hvr commented on May 20, 2017

   

   hvr

   on May 20, 2017

   Member

   More actions

   fwiw, the plan is to inject the CDN download count data into Hackage so it can provide more reliably counts again; I just need to finish the S3-logic to reliably fetch and aggregate the daily data.
5. gbaz

   added

   enhancement

   on Sep 30, 2017
6. gbaz commented on Mar 23, 2018

   

   gbaz

   on Mar 23, 2018

   Contributor

   More actions

   We have a plan to tackle the CDN issue so that shouldn't be a blocker on this.
7. brandonchinn178 commented on Nov 22, 2021

   

   brandonchinn178

   on Nov 22, 2021

   Last edited by brandonchinn178

   Contributor

   More actions

   👍 /packages/top orders packages by number of downloads in the past 30 days, but it'd be nice even if it supported a query parameter to toggle all-time downloads.

   I'm writing a script to run on the X most popular packages (by number of downloads), and /packages/top/ provides a decent proxy, but it'd be better to work off the all-time stats.

   (also, /packages/top doesn't have a JSON option, so I'm having to scrape HTML right now. Can this endpoint also allow returning JSON format?)
8. noughtmare commented on Mar 8, 2025

   

   noughtmare

   on Mar 8, 2025

   More actions

   Just now, I wanted to have the number of downloads for a list of packages (discussed here).

   I resorted to writing this script:

   while read -r line; do
     echo -ne "$line\t"
     curl "https://hackage.haskell.org/package/$line" 2>/dev/null | rg "(\d+) in the last 30 days" -r '$1' -o
   done

   That's obviously not very friendly to the server, so I hope an endpoint could provide this functionality with a single request in the future.
9. noughtmare commented on Mar 8, 2025

   

   noughtmare

   on Mar 8, 2025

   More actions

   I guess the existing https://hackage.haskell.org/packages/top endpoint could serve this purpose. So, I guess this is friendlier:

   top=$(curl -H "Accept: application/json" https://hackage.haskell.org/packages/top)
   while read -r line; do
     echo -ne "$line\t"
     echo "$top" | jq ".[] | select(.packageName == \"$line\") | .downloads"
   done