Skip to content

users.messages.list and users.threads.list does NOT return messages/threads in reverse-chronological order #3508

Closed

Description

@wdanilo

Description

users.messages.list returns messages in a little bit random order.
I also reported the issue here, but knowing Google's issue tracker, I will get a sensible reply there in a few years, so trying my luck here as well :(

Code

I'm using the following code:

async test() {
    console.log(
      "Performing test if 'users.messages.list' returns messages sorted by decreasing dates and historyIDs.",
    )
    const count = 100
    const cfg: GmailApi.Params$Resource$Users$Messages$List = {
      userId: 'me',
      maxResults: count,
    }
    const resp = await Auth.catchAllOAuth(() => this.api.users.messages.list(cfg))
    if (resp.isErr()) {
      console.error('GMail API query error.', resp.error)
      return
    }
    const list = resp.value.data.messages
    if (list == null) {
      console.error('GMail API invalid response.')
      return
    }
    let isHistoryIdDecreasing = true
    let isDateDecreasing = true
    let prevHistoryId: null | number = null
    let prevDate: null | number = null
    let i = 0
    for (const message of list) {
      i++
      const id = message.id
      if (id == null) {
        console.error('Message is missing the "id" field.')
        return
      }
      const msgQuery = {userId: 'me', id}
      const msgResp = await Auth.catchAllOAuth(() => this.api.users.messages.get(msgQuery))
      if (msgResp.isErr()) {
        console.error('GMail API query error.', msgResp.error)
        return
      }
      const historyIdStr = msgResp.value.data.historyId
      const dateStr = msgResp.value.data.internalDate
      if (!historyIdStr || !dateStr) {
        console.error('Message is missing the "historyId" or "internalDate" field.')
        return
      }
      const historyId = parseInt(historyIdStr, 10)
      const date = parseInt(dateStr, 10)
      let localIsHistoryIdDecreasing = true
      let localIsDateDecreasing = true
      if (prevHistoryId != null && prevHistoryId < historyId) {
        isHistoryIdDecreasing = false
        localIsHistoryIdDecreasing = false
      }
      if (prevDate != null && prevDate < date) {
        isDateDecreasing = false
        localIsDateDecreasing = false
      }
      prevHistoryId = historyId
      prevDate = date
      console.log(
        `${i}/${count}: historyId: ${historyId}, date: ${date}, isHistoryIdDecreasing: ${localIsHistoryIdDecreasing}, isDateDecreasing: ${localIsDateDecreasing}`,
      )
    }
    console.log({isHistoryIdDecreasing, isDateDecreasing})
  }

Output

Performing test if 'users.messages.list' returns messages sorted by decreasing dates and historyIDs.
1/100: historyId: 15297863, date: 1719191761000, isHistoryIdDecreasing: true, isDateDecreasing: true
2/100: historyId: 15297367, date: 1719170805000, isHistoryIdDecreasing: true, isDateDecreasing: true
3/100: historyId: 15297189, date: 1719159861000, isHistoryIdDecreasing: true, isDateDecreasing: true
4/100: historyId: 15297049, date: 1719150301000, isHistoryIdDecreasing: true, isDateDecreasing: true
5/100: historyId: 15297132, date: 1719149574000, isHistoryIdDecreasing: false, isDateDecreasing: true
6/100: historyId: 15296597, date: 1719147670000, isHistoryIdDecreasing: true, isDateDecreasing: true
7/100: historyId: 15297036, date: 1719122788000, isHistoryIdDecreasing: false, isDateDecreasing: true
8/100: historyId: 15296855, date: 1719120755000, isHistoryIdDecreasing: true, isDateDecreasing: true
9/100: historyId: 15296852, date: 1719120714000, isHistoryIdDecreasing: true, isDateDecreasing: true
10/100: historyId: 15297511, date: 1719118502000, isHistoryIdDecreasing: false, isDateDecreasing: true
11/100: historyId: 15296422, date: 1719108184000, isHistoryIdDecreasing: true, isDateDecreasing: true
12/100: historyId: 15296007, date: 1719103434000, isHistoryIdDecreasing: true, isDateDecreasing: true
13/100: historyId: 15297035, date: 1719094619000, isHistoryIdDecreasing: false, isDateDecreasing: true
14/100: historyId: 15297035, date: 1719094618000, isHistoryIdDecreasing: true, isDateDecreasing: true
15/100: historyId: 15295930, date: 1719088891000, isHistoryIdDecreasing: true, isDateDecreasing: true
16/100: historyId: 15295489, date: 1719077834000, isHistoryIdDecreasing: true, isDateDecreasing: true
17/100: historyId: 15297034, date: 1719077423000, isHistoryIdDecreasing: false, isDateDecreasing: true
18/100: historyId: 15295374, date: 1719076577000, isHistoryIdDecreasing: true, isDateDecreasing: true
19/100: historyId: 15297544, date: 1719070375000, isHistoryIdDecreasing: false, isDateDecreasing: true
20/100: historyId: 15297032, date: 1719069634000, isHistoryIdDecreasing: true, isDateDecreasing: true
21/100: historyId: 15297569, date: 1719058421000, isHistoryIdDecreasing: false, isDateDecreasing: true
22/100: historyId: 15295569, date: 1719057377000, isHistoryIdDecreasing: true, isDateDecreasing: true
23/100: historyId: 15297030, date: 1719053485000, isHistoryIdDecreasing: false, isDateDecreasing: true
24/100: historyId: 15295971, date: 1719044954000, isHistoryIdDecreasing: true, isDateDecreasing: true
25/100: historyId: 15297029, date: 1719039969000, isHistoryIdDecreasing: false, isDateDecreasing: true
26/100: historyId: 15297429, date: 1719039871000, isHistoryIdDecreasing: false, isDateDecreasing: true
27/100: historyId: 15297027, date: 1719033506000, isHistoryIdDecreasing: true, isDateDecreasing: true
28/100: historyId: 15294309, date: 1719031329000, isHistoryIdDecreasing: true, isDateDecreasing: true
29/100: historyId: 15295111, date: 1719027583000, isHistoryIdDecreasing: false, isDateDecreasing: true
30/100: historyId: 15297593, date: 1719026659000, isHistoryIdDecreasing: false, isDateDecreasing: true
31/100: historyId: 15297025, date: 1719025075000, isHistoryIdDecreasing: true, isDateDecreasing: true
32/100: historyId: 15297024, date: 1719024799000, isHistoryIdDecreasing: true, isDateDecreasing: true
33/100: historyId: 15297618, date: 1719024388000, isHistoryIdDecreasing: false, isDateDecreasing: true
34/100: historyId: 15297022, date: 1719021288000, isHistoryIdDecreasing: true, isDateDecreasing: true
35/100: historyId: 15297642, date: 1719021206000, isHistoryIdDecreasing: false, isDateDecreasing: true
36/100: historyId: 15293469, date: 1719021069000, isHistoryIdDecreasing: true, isDateDecreasing: true
37/100: historyId: 15297642, date: 1719021008000, isHistoryIdDecreasing: false, isDateDecreasing: true
38/100: historyId: 15297671, date: 1719008653000, isHistoryIdDecreasing: false, isDateDecreasing: true
39/100: historyId: 15293217, date: 1719008487000, isHistoryIdDecreasing: true, isDateDecreasing: true
40/100: historyId: 15297705, date: 1719008444000, isHistoryIdDecreasing: false, isDateDecreasing: true
41/100: historyId: 15293049, date: 1719007821000, isHistoryIdDecreasing: true, isDateDecreasing: true
42/100: historyId: 15292962, date: 1719005719000, isHistoryIdDecreasing: true, isDateDecreasing: true
43/100: historyId: 15297739, date: 1719004759000, isHistoryIdDecreasing: false, isDateDecreasing: true
44/100: historyId: 15293296, date: 1719004141000, isHistoryIdDecreasing: true, isDateDecreasing: true
45/100: historyId: 15292645, date: 1719002715000, isHistoryIdDecreasing: true, isDateDecreasing: true
46/100: historyId: 15292586, date: 1719001722000, isHistoryIdDecreasing: true, isDateDecreasing: true
47/100: historyId: 15292507, date: 1718994153000, isHistoryIdDecreasing: true, isDateDecreasing: true
48/100: historyId: 15296270, date: 1719000410000, isHistoryIdDecreasing: false, isDateDecreasing: false
49/100: historyId: 15296304, date: 1719000210000, isHistoryIdDecreasing: false, isDateDecreasing: true
50/100: historyId: 15292265, date: 1719000212000, isHistoryIdDecreasing: true, isDateDecreasing: false
51/100: historyId: 15293696, date: 1719000158000, isHistoryIdDecreasing: false, isDateDecreasing: true
52/100: historyId: 15292133, date: 1719000132000, isHistoryIdDecreasing: true, isDateDecreasing: true
53/100: historyId: 15291968, date: 1718999917000, isHistoryIdDecreasing: true, isDateDecreasing: true
54/100: historyId: 15291886, date: 1718999562000, isHistoryIdDecreasing: true, isDateDecreasing: true
55/100: historyId: 15291826, date: 1718999475000, isHistoryIdDecreasing: true, isDateDecreasing: true
56/100: historyId: 15291744, date: 1718999417000, isHistoryIdDecreasing: true, isDateDecreasing: true
57/100: historyId: 15291662, date: 1718999355000, isHistoryIdDecreasing: true, isDateDecreasing: true
58/100: historyId: 15291604, date: 1718999152000, isHistoryIdDecreasing: true, isDateDecreasing: true
59/100: historyId: 15291544, date: 1718999072000, isHistoryIdDecreasing: true, isDateDecreasing: true
60/100: historyId: 15291461, date: 1718998798000, isHistoryIdDecreasing: true, isDateDecreasing: true
61/100: historyId: 15291384, date: 1718998480000, isHistoryIdDecreasing: true, isDateDecreasing: true
62/100: historyId: 15291302, date: 1718998079000, isHistoryIdDecreasing: true, isDateDecreasing: true
63/100: historyId: 15291242, date: 1718997877000, isHistoryIdDecreasing: true, isDateDecreasing: true
64/100: historyId: 15291164, date: 1718997874000, isHistoryIdDecreasing: true, isDateDecreasing: true
65/100: historyId: 15291101, date: 1718997051000, isHistoryIdDecreasing: true, isDateDecreasing: true
66/100: historyId: 15291019, date: 1718997022000, isHistoryIdDecreasing: true, isDateDecreasing: true
67/100: historyId: 15290937, date: 1718996810000, isHistoryIdDecreasing: true, isDateDecreasing: true
68/100: historyId: 15290879, date: 1718996476000, isHistoryIdDecreasing: true, isDateDecreasing: true
69/100: historyId: 15290819, date: 1718996035000, isHistoryIdDecreasing: true, isDateDecreasing: true
70/100: historyId: 15290759, date: 1718995732000, isHistoryIdDecreasing: true, isDateDecreasing: true
71/100: historyId: 15290677, date: 1718995321000, isHistoryIdDecreasing: true, isDateDecreasing: true
72/100: historyId: 15290621, date: 1718995295000, isHistoryIdDecreasing: true, isDateDecreasing: true
73/100: historyId: 15290510, date: 1718994605000, isHistoryIdDecreasing: true, isDateDecreasing: true
74/100: historyId: 15290450, date: 1718994479000, isHistoryIdDecreasing: true, isDateDecreasing: true
75/100: historyId: 15290371, date: 1718994196000, isHistoryIdDecreasing: true, isDateDecreasing: true
76/100: historyId: 15290592, date: 1718994167000, isHistoryIdDecreasing: false, isDateDecreasing: true
77/100: historyId: 15290141, date: 1718993384000, isHistoryIdDecreasing: true, isDateDecreasing: true
78/100: historyId: 15290059, date: 1718993022000, isHistoryIdDecreasing: true, isDateDecreasing: true
79/100: historyId: 15289999, date: 1718992719000, isHistoryIdDecreasing: true, isDateDecreasing: true
80/100: historyId: 15289917, date: 1718992717000, isHistoryIdDecreasing: true, isDateDecreasing: true
81/100: historyId: 15289836, date: 1718992611000, isHistoryIdDecreasing: true, isDateDecreasing: true
82/100: historyId: 15289755, date: 1718992550000, isHistoryIdDecreasing: true, isDateDecreasing: true
83/100: historyId: 15289673, date: 1718992117000, isHistoryIdDecreasing: true, isDateDecreasing: true
84/100: historyId: 15289614, date: 1718992106000, isHistoryIdDecreasing: true, isDateDecreasing: true
85/100: historyId: 15289532, date: 1718992105000, isHistoryIdDecreasing: true, isDateDecreasing: true
86/100: historyId: 15289468, date: 1718991762000, isHistoryIdDecreasing: true, isDateDecreasing: true
87/100: historyId: 15289402, date: 1718991761000, isHistoryIdDecreasing: true, isDateDecreasing: true
88/100: historyId: 15289338, date: 1718991702000, isHistoryIdDecreasing: true, isDateDecreasing: true
89/100: historyId: 15289249, date: 1718991700000, isHistoryIdDecreasing: true, isDateDecreasing: true
90/100: historyId: 15289156, date: 1718991369000, isHistoryIdDecreasing: true, isDateDecreasing: true
91/100: historyId: 15297775, date: 1718991164000, isHistoryIdDecreasing: false, isDateDecreasing: true
92/100: historyId: 15297396, date: 1718991124000, isHistoryIdDecreasing: true, isDateDecreasing: true
93/100: historyId: 15297463, date: 1718989132000, isHistoryIdDecreasing: false, isDateDecreasing: true
94/100: historyId: 15297463, date: 1718989129000, isHistoryIdDecreasing: true, isDateDecreasing: true
95/100: historyId: 15288682, date: 1718987737000, isHistoryIdDecreasing: true, isDateDecreasing: true
96/100: historyId: 15288603, date: 1718986405000, isHistoryIdDecreasing: true, isDateDecreasing: true
97/100: historyId: 15288488, date: 1718982653000, isHistoryIdDecreasing: true, isDateDecreasing: true
98/100: historyId: 15288464, date: 1718982414000, isHistoryIdDecreasing: true, isDateDecreasing: true
99/100: historyId: 15288464, date: 1718982411000, isHistoryIdDecreasing: true, isDateDecreasing: true
100/100: historyId: 15297373, date: 1718980504000, isHistoryIdDecreasing: false, isDateDecreasing: true
{ isHistoryIdDecreasing: false, isDateDecreasing: false }

Summary

As you can see, the message internalDate is sometimes ... increasing. I've observed the same behavior for users.threads.list. In such a scenario:

  1. What is described here: https://developers.google.com/gmail/api/guides/sync is not true, I can't query for the first message of users.messages.list, remember its historyId and assume this is the newest historyId, as this might NOT be the newest message.

  2. What is described here: https://developers.google.com/gmail/api/reference/rest/v1/users.history/list (History IDs increase chronologically but are not contiguous with random gaps in between valid IDs) is not true, as you can see, historyIDs are sometimes increasing for older messages.

Activity

  1. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Jun 24, 2024
  2. wdanilo commented on Jun 24, 2024

    @wdanilo
    Author

    Also, here is a code checking the same broken behavior for users.threads.list:

    Code

    async test2() {
        console.log(
          "Performing test if 'users.threads.list' returns threads sorted by decreasing dates and historyIDs of their first messages.",
        )
        const count = 100
        const cfg: GmailApi.Params$Resource$Users$Threads$List = {
          userId: 'me',
          maxResults: count,
        }
        const resp = await Auth.catchAllOAuth(() => this.api.users.threads.list(cfg))
        if (resp.isErr()) {
          console.error('GMail API query error.', resp.error)
          return
        }
        const list = resp.value.data.threads
        if (list == null) {
          console.error('GMail API invalid response.')
          return
        }
        let isHistoryIdDecreasing = true
        let isDateDecreasing = true
        let prevHistoryId: null | number = null
        let prevDate: null | number = null
        let i = 0
        for (const thread of list) {
          i++
          const id = thread.id
          if (id == null) {
            console.error('Thread is missing the "id" field.')
            return
          }
          const threadQuery = {userId: 'me', id}
          const threadResp = await Auth.catchAllOAuth(() => this.api.users.threads.get(threadQuery))
          if (threadResp.isErr()) {
            console.error('GMail API query error.', threadResp.error)
            return
          }
          if (threadResp.value.data.messages == null) {
            console.error('Thread is missing the "messages" field.')
            return
          }
          const firstMessage = threadResp.value.data.messages[0]
          if (firstMessage == null) {
            console.error('Thread is missing the first message.')
            return
          }
          const historyIdStr = firstMessage.historyId
          const dateStr = firstMessage.internalDate
          if (!historyIdStr || !dateStr) {
            console.error('First message is missing the "historyId" or "internalDate" field.')
            return
          }
          const historyId = parseInt(historyIdStr, 10)
          const date = parseInt(dateStr, 10)
          let localIsHistoryIdDecreasing = true
          let localIsDateDecreasing = true
          if (prevHistoryId != null && prevHistoryId < historyId) {
            isHistoryIdDecreasing = false
            localIsHistoryIdDecreasing = false
          }
          if (prevDate != null && prevDate < date) {
            isDateDecreasing = false
            localIsDateDecreasing = false
          }
          prevHistoryId = historyId
          prevDate = date
          console.log(
            `${i}/${count}: historyId: ${historyId}, date: ${date}, isHistoryIdDecreasing: ${localIsHistoryIdDecreasing}, isDateDecreasing: ${localIsDateDecreasing}`,
          )
        }
        console.log({isHistoryIdDecreasing, isDateDecreasing})
      }

    Output:

    Performing test if 'users.threads.list' returns threads sorted by decreasing dates and historyIDs of their first messages.
    1/100: historyId: 15297863, date: 1719191761000, isHistoryIdDecreasing: true, isDateDecreasing: true
    2/100: historyId: 15297367, date: 1719170805000, isHistoryIdDecreasing: true, isDateDecreasing: true
    3/100: historyId: 15297189, date: 1719159861000, isHistoryIdDecreasing: true, isDateDecreasing: true
    4/100: historyId: 15297049, date: 1719150301000, isHistoryIdDecreasing: true, isDateDecreasing: true
    5/100: historyId: 15297132, date: 1719149574000, isHistoryIdDecreasing: false, isDateDecreasing: true
    6/100: historyId: 15296597, date: 1719147670000, isHistoryIdDecreasing: true, isDateDecreasing: true
    7/100: historyId: 15297036, date: 1719122788000, isHistoryIdDecreasing: false, isDateDecreasing: true
    8/100: historyId: 15296855, date: 1719120755000, isHistoryIdDecreasing: true, isDateDecreasing: true
    9/100: historyId: 15297511, date: 1719118502000, isHistoryIdDecreasing: false, isDateDecreasing: true
    10/100: historyId: 15295111, date: 1719027583000, isHistoryIdDecreasing: true, isDateDecreasing: true
    11/100: historyId: 15296007, date: 1719103434000, isHistoryIdDecreasing: false, isDateDecreasing: false
    12/100: historyId: 15297035, date: 1719094618000, isHistoryIdDecreasing: false, isDateDecreasing: true
    13/100: historyId: 15295569, date: 1719057377000, isHistoryIdDecreasing: true, isDateDecreasing: true
    14/100: historyId: 15295489, date: 1719077834000, isHistoryIdDecreasing: true, isDateDecreasing: false
    15/100: historyId: 15297034, date: 1719077423000, isHistoryIdDecreasing: false, isDateDecreasing: true
    16/100: historyId: 15295374, date: 1719076577000, isHistoryIdDecreasing: true, isDateDecreasing: true
    17/100: historyId: 15297544, date: 1719070375000, isHistoryIdDecreasing: false, isDateDecreasing: true
    18/100: historyId: 15297032, date: 1719069634000, isHistoryIdDecreasing: true, isDateDecreasing: true
    19/100: historyId: 15297569, date: 1719058421000, isHistoryIdDecreasing: false, isDateDecreasing: true
    20/100: historyId: 15297030, date: 1719053485000, isHistoryIdDecreasing: true, isDateDecreasing: true
    21/100: historyId: 15295971, date: 1719044954000, isHistoryIdDecreasing: true, isDateDecreasing: true
    22/100: historyId: 15297029, date: 1719039969000, isHistoryIdDecreasing: false, isDateDecreasing: true
    23/100: historyId: 15297429, date: 1719039871000, isHistoryIdDecreasing: false, isDateDecreasing: true
    24/100: historyId: 15297027, date: 1719033506000, isHistoryIdDecreasing: true, isDateDecreasing: true
    25/100: historyId: 15294309, date: 1719031329000, isHistoryIdDecreasing: true, isDateDecreasing: true
    26/100: historyId: 15297593, date: 1719026659000, isHistoryIdDecreasing: false, isDateDecreasing: true
    27/100: historyId: 15297025, date: 1719025075000, isHistoryIdDecreasing: true, isDateDecreasing: true
    28/100: historyId: 15297024, date: 1719024799000, isHistoryIdDecreasing: true, isDateDecreasing: true
    29/100: historyId: 15297618, date: 1719024388000, isHistoryIdDecreasing: false, isDateDecreasing: true
    30/100: historyId: 15297022, date: 1719021288000, isHistoryIdDecreasing: true, isDateDecreasing: true
    31/100: historyId: 15297642, date: 1719021008000, isHistoryIdDecreasing: false, isDateDecreasing: true
    32/100: historyId: 15293469, date: 1719021069000, isHistoryIdDecreasing: true, isDateDecreasing: false
    33/100: historyId: 15297671, date: 1719008653000, isHistoryIdDecreasing: false, isDateDecreasing: true
    34/100: historyId: 15293217, date: 1719008487000, isHistoryIdDecreasing: true, isDateDecreasing: true
    35/100: historyId: 15297705, date: 1719008444000, isHistoryIdDecreasing: false, isDateDecreasing: true
    36/100: historyId: 15293049, date: 1719007821000, isHistoryIdDecreasing: true, isDateDecreasing: true
    37/100: historyId: 15292962, date: 1719005719000, isHistoryIdDecreasing: true, isDateDecreasing: true
    38/100: historyId: 15297739, date: 1719004759000, isHistoryIdDecreasing: false, isDateDecreasing: true
    39/100: historyId: 15293296, date: 1719004141000, isHistoryIdDecreasing: true, isDateDecreasing: true
    40/100: historyId: 15292645, date: 1719002715000, isHistoryIdDecreasing: true, isDateDecreasing: true
    41/100: historyId: 15290059, date: 1718993022000, isHistoryIdDecreasing: true, isDateDecreasing: true
    42/100: historyId: 15292507, date: 1718994153000, isHistoryIdDecreasing: false, isDateDecreasing: false
    43/100: historyId: 15296270, date: 1719000410000, isHistoryIdDecreasing: false, isDateDecreasing: false
    44/100: historyId: 15296304, date: 1719000210000, isHistoryIdDecreasing: false, isDateDecreasing: true
    45/100: historyId: 15291662, date: 1718999355000, isHistoryIdDecreasing: true, isDateDecreasing: true
    46/100: historyId: 15293696, date: 1719000158000, isHistoryIdDecreasing: false, isDateDecreasing: false
    47/100: historyId: 15292133, date: 1719000132000, isHistoryIdDecreasing: true, isDateDecreasing: true
    48/100: historyId: 15291968, date: 1718999917000, isHistoryIdDecreasing: true, isDateDecreasing: true
    49/100: historyId: 15291886, date: 1718999562000, isHistoryIdDecreasing: true, isDateDecreasing: true
    50/100: historyId: 15291744, date: 1718999417000, isHistoryIdDecreasing: true, isDateDecreasing: true
    51/100: historyId: 15291461, date: 1718998798000, isHistoryIdDecreasing: true, isDateDecreasing: true
    52/100: historyId: 15291384, date: 1718998480000, isHistoryIdDecreasing: true, isDateDecreasing: true
    53/100: historyId: 15291302, date: 1718998079000, isHistoryIdDecreasing: true, isDateDecreasing: true
    54/100: historyId: 15291164, date: 1718997874000, isHistoryIdDecreasing: true, isDateDecreasing: true
    55/100: historyId: 15291019, date: 1718997022000, isHistoryIdDecreasing: true, isDateDecreasing: true
    56/100: historyId: 15290937, date: 1718996810000, isHistoryIdDecreasing: true, isDateDecreasing: true
    57/100: historyId: 15290677, date: 1718995321000, isHistoryIdDecreasing: true, isDateDecreasing: true
    58/100: historyId: 15290510, date: 1718994605000, isHistoryIdDecreasing: true, isDateDecreasing: true
    59/100: historyId: 15290371, date: 1718994196000, isHistoryIdDecreasing: true, isDateDecreasing: true
    60/100: historyId: 15290592, date: 1718994167000, isHistoryIdDecreasing: false, isDateDecreasing: true
    61/100: historyId: 15290141, date: 1718993384000, isHistoryIdDecreasing: true, isDateDecreasing: true
    62/100: historyId: 15289917, date: 1718992717000, isHistoryIdDecreasing: true, isDateDecreasing: true
    63/100: historyId: 15289836, date: 1718992611000, isHistoryIdDecreasing: true, isDateDecreasing: true
    64/100: historyId: 15289755, date: 1718992550000, isHistoryIdDecreasing: true, isDateDecreasing: true
    65/100: historyId: 15289673, date: 1718992117000, isHistoryIdDecreasing: true, isDateDecreasing: true
    66/100: historyId: 15289532, date: 1718992105000, isHistoryIdDecreasing: true, isDateDecreasing: true
    67/100: historyId: 15289249, date: 1718991700000, isHistoryIdDecreasing: true, isDateDecreasing: true
    68/100: historyId: 15289156, date: 1718991369000, isHistoryIdDecreasing: true, isDateDecreasing: true
    69/100: historyId: 15297775, date: 1718991164000, isHistoryIdDecreasing: false, isDateDecreasing: true
    70/100: historyId: 15297396, date: 1718991124000, isHistoryIdDecreasing: true, isDateDecreasing: true
    71/100: historyId: 15297463, date: 1718989129000, isHistoryIdDecreasing: false, isDateDecreasing: true
    72/100: historyId: 15288682, date: 1718987737000, isHistoryIdDecreasing: true, isDateDecreasing: true
    73/100: historyId: 15288603, date: 1718986405000, isHistoryIdDecreasing: true, isDateDecreasing: true
    74/100: historyId: 15288488, date: 1718982653000, isHistoryIdDecreasing: true, isDateDecreasing: true
    75/100: historyId: 15288464, date: 1718982411000, isHistoryIdDecreasing: true, isDateDecreasing: true
    76/100: historyId: 15297373, date: 1718980504000, isHistoryIdDecreasing: false, isDateDecreasing: true
    77/100: historyId: 15288268, date: 1718978327000, isHistoryIdDecreasing: true, isDateDecreasing: true
    78/100: historyId: 15288267, date: 1718971574000, isHistoryIdDecreasing: true, isDateDecreasing: true
    79/100: historyId: 15293754, date: 1718909485000, isHistoryIdDecreasing: false, isDateDecreasing: true
    80/100: historyId: 15287783, date: 1718956922000, isHistoryIdDecreasing: true, isDateDecreasing: false
    81/100: historyId: 15287862, date: 1718956695000, isHistoryIdDecreasing: false, isDateDecreasing: true
    82/100: historyId: 15288266, date: 1718954346000, isHistoryIdDecreasing: false, isDateDecreasing: true
    83/100: historyId: 15287564, date: 1718953202000, isHistoryIdDecreasing: true, isDateDecreasing: true
    84/100: historyId: 15287478, date: 1718935413000, isHistoryIdDecreasing: true, isDateDecreasing: true
    85/100: historyId: 15287253, date: 1718924448000, isHistoryIdDecreasing: true, isDateDecreasing: true
    86/100: historyId: 15287477, date: 1718920072000, isHistoryIdDecreasing: false, isDateDecreasing: true
    87/100: historyId: 15287061, date: 1718917993000, isHistoryIdDecreasing: true, isDateDecreasing: true
    88/100: historyId: 15286956, date: 1718909913000, isHistoryIdDecreasing: true, isDateDecreasing: true
    89/100: historyId: 15286786, date: 1718904398000, isHistoryIdDecreasing: true, isDateDecreasing: true
    90/100: historyId: 15286705, date: 1718898024000, isHistoryIdDecreasing: true, isDateDecreasing: true
    91/100: historyId: 15286624, date: 1718900423000, isHistoryIdDecreasing: true, isDateDecreasing: false
    92/100: historyId: 15295890, date: 1718857229000, isHistoryIdDecreasing: false, isDateDecreasing: true
    93/100: historyId: 15286444, date: 1718883913000, isHistoryIdDecreasing: true, isDateDecreasing: false
    94/100: historyId: 15287475, date: 1718879255000, isHistoryIdDecreasing: false, isDateDecreasing: true
    95/100: historyId: 15286284, date: 1718876932000, isHistoryIdDecreasing: true, isDateDecreasing: true
    96/100: historyId: 15287493, date: 1718859740000, isHistoryIdDecreasing: false, isDateDecreasing: true
    97/100: historyId: 15286009, date: 1718853779000, isHistoryIdDecreasing: true, isDateDecreasing: true
    98/100: historyId: 15285920, date: 1718839427000, isHistoryIdDecreasing: true, isDateDecreasing: true
    99/100: historyId: 15285840, date: 1718831857000, isHistoryIdDecreasing: true, isDateDecreasing: true
    100/100: historyId: 15285787, date: 1718829287000, isHistoryIdDecreasing: true, isDateDecreasing: true
    { isHistoryIdDecreasing: false, isDateDecreasing: false }
    
  3. wdanilo commented on Jun 24, 2024

    @wdanilo
    Author

    Also, to make it more visual, I plotted the time I get from users.messages.list. As you can see, there is a drop, so we got 2 messages with date way earlier than messages we get after it. X-axis is the message # from the users.messages.list. Y-axis is its internalDate:

    image

    And here is plot of historyId. X-axis is the message # from the users.messages.list. Y-axis is its historyId. So, in contrast to what Google docs state, historyId does not increase chronologically.

    image
  4. wdanilo commented on Jun 25, 2024

    @wdanilo
    Author

    What is even worse, it appears that users.history.list does not provide results in chronological order neither, despite Google claims so (https://developers.google.com/gmail/api/reference/rest/v1/users.history/list) (citation: Lists the history of all changes to the given mailbox. History results are returned in chronological order (increasing historyId).). Below is a code to test it:

    Code

    async test_invariants_history_list() {
        console.log(
          "Performing test if 'users.history.list' returns history records sorted by increasing update date.",
        )
        const viz = (t: boolean) => (t ? '✅' : '❌')
        const count = 100
        const cfg: GmailApi.Params$Resource$Users$Messages$List = {
          userId: 'me',
          maxResults: count,
        }
        const resp = await Auth.catchAllOAuth(() => this.api.users.messages.list(cfg))
        if (resp.isErr()) {
          console.error('GMail API query error.', resp.error)
          return
        }
        const list = resp.value.data.messages
        if (list == null) {
          console.error('GMail API invalid response.')
          return
        }
        const lastMessage = list[list.length - 1]
        if (lastMessage == null) {
          console.error('Last message is missing.')
          return
        }
        const lastMessageId = lastMessage.id
        if (lastMessageId == null) {
          console.error('Last message is missing the "id" field.')
          return
        }
        const lastMessageQuery = {userId: 'me', id: lastMessageId}
        const lastMessageResp = await Auth.catchAllOAuth(() =>
          this.api.users.messages.get(lastMessageQuery),
        )
        if (lastMessageResp.isErr()) {
          console.error('GMail API query error.', lastMessageResp.error)
          return
        }
        const startHistoryId = lastMessageResp.value.data.historyId
        if (startHistoryId == null) {
          console.error('Last message is missing the "historyId" field.')
          return
        }
        const historyCfg: GmailApi.Params$Resource$Users$History$List = {
          userId: 'me',
          startHistoryId,
          maxResults: 500,
        }
        const historyResp = await Auth.catchAllOAuth(() => this.api.users.history.list(historyCfg))
        if (historyResp.isErr()) {
          console.error('GMail API query error.', historyResp.error)
          return
        }
        const historyList = historyResp.value.data.history
        if (historyList == null) {
          console.error('GMail API invalid response.')
          return
        }
    
        let localDateIncreasing = true
        let prevDate: null | number = null
        const updateCount = historyList.length
        let i = 0
        for (const update of historyList) {
          i++
          if (update.messages == null) {
            console.error('Update is missing the "messages" field.')
            return
          }
          let m = 0
          for (const message of update.messages) {
            m++
            if (message.id == null) {
              console.error('Message is missing the "id" field.')
              return
            }
            const messageQuery = {userId: 'me', id: message.id}
            const messageResp = await Auth.catchAllOAuth(() =>
              this.api.users.messages.get(messageQuery),
            )
            if (messageResp.isErr()) {
              console.error('GMail API query error.', messageResp.error)
              continue
            }
            const dateStr = messageResp.value.data.internalDate
            if (!dateStr) {
              console.error('Message is missing the "internalDate" field.')
              return
            }
            const date = parseInt(dateStr, 10)
            if (prevDate != null && prevDate > date) {
              localDateIncreasing = false
            } else {
              localDateIncreasing = true
            }
            prevDate = date
            console.log(
              `${i}/${updateCount}[${m}]: date: ${date}, dateIncreasing: ${viz(localDateIncreasing)}`,
            )
          }
        }
      }

    Output

    1/232[1]: date: 1718992106000, dateIncreasing: ✅
    2/232[1]: date: 1718992117000, dateIncreasing: ✅
    3/232[1]: date: 1718992550000, dateIncreasing: ✅
    4/232[1]: date: 1718992611000, dateIncreasing: ✅
    5/232[1]: date: 1718992717000, dateIncreasing: ✅
    6/232[1]: date: 1718992719000, dateIncreasing: ✅
    7/232[1]: date: 1718993022000, dateIncreasing: ✅
    8/232[1]: date: 1718993384000, dateIncreasing: ✅
    9/232[1]: date: 1718994167000, dateIncreasing: ✅
    10/232[1]: date: 1718994167000, dateIncreasing: ✅
    11/232[1]: date: 1718994167000, dateIncreasing: ✅
    12/232[1]: date: 1718994167000, dateIncreasing: ✅
    13/232[1]: date: 1718994196000, dateIncreasing: ✅
    14/232[1]: date: 1718994479000, dateIncreasing: ✅
    15/232[1]: date: 1718994605000, dateIncreasing: ✅
    16/232[1]: date: 1718994167000, dateIncreasing: ❌
    17/232[1]: date: 1718995295000, dateIncreasing: ✅
    18/232[1]: date: 1718995321000, dateIncreasing: ✅
    19/232[1]: date: 1718995732000, dateIncreasing: ✅
    20/232[1]: date: 1718996035000, dateIncreasing: ✅
    21/232[1]: date: 1718996476000, dateIncreasing: ✅
    22/232[1]: date: 1718996810000, dateIncreasing: ✅
    23/232[1]: date: 1718997022000, dateIncreasing: ✅
    24/232[1]: date: 1718997051000, dateIncreasing: ✅
    25/232[1]: date: 1718997874000, dateIncreasing: ✅
    26/232[1]: date: 1718997877000, dateIncreasing: ✅
    27/232[1]: date: 1718998079000, dateIncreasing: ✅
    28/232[1]: date: 1718998480000, dateIncreasing: ✅
    29/232[1]: date: 1718998798000, dateIncreasing: ✅
    30/232[1]: date: 1718999072000, dateIncreasing: ✅
    31/232[1]: date: 1718999152000, dateIncreasing: ✅
    32/232[1]: date: 1718999355000, dateIncreasing: ✅
    33/232[1]: date: 1718999417000, dateIncreasing: ✅
    34/232[1]: date: 1718999475000, dateIncreasing: ✅
    35/232[1]: date: 1718999562000, dateIncreasing: ✅
    36/232[1]: date: 1718999917000, dateIncreasing: ✅
    37/232[1]: date: 1719000132000, dateIncreasing: ✅
    38/232[1]: date: 1719000132000, dateIncreasing: ✅
    39/232[1]: date: 1719000132000, dateIncreasing: ✅
    40/232[1]: date: 1719000132000, dateIncreasing: ✅
    41/232[1]: date: 1719000132000, dateIncreasing: ✅
    42/232[1]: date: 1719000132000, dateIncreasing: ✅
    43/232[1]: date: 1719000158000, dateIncreasing: ✅
    44/232[1]: date: 1719000212000, dateIncreasing: ✅
    45/232[1]: date: 1719000210000, dateIncreasing: ❌
    GMail API query error. {
      level: 4,
      message: 'OAuth call failed.',
      data: {
        error: GaxiosError: Requested entity was not found.
            at Gaxios._request (<redacted>)
            at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
            at async UserRefreshClient.requestAsync (<redacted>)
            at async catchAllOAuth (<redacted>)
            at async Sync.test_invariants_history_list (<redacted>)
          response: [Object],
          config: [Object],
          code: 404,
          errors: [Array]
        }
      },
      flags: {}
    }
    47/232[1]: date: 1719000410000, dateIncreasing: ✅
    48/232[1]: date: 1718994153000, dateIncreasing: ❌
    49/232[1]: date: 1719001722000, dateIncreasing: ✅
    50/232[1]: date: 1719002715000, dateIncreasing: ✅
    51/232[1]: date: 1719004141000, dateIncreasing: ✅
    52/232[1]: date: 1719004141000, dateIncreasing: ✅
    53/232[1]: date: 1719004141000, dateIncreasing: ✅
    54/232[1]: date: 1719004141000, dateIncreasing: ✅
    55/232[1]: date: 1719004759000, dateIncreasing: ✅
    56/232[1]: date: 1719005719000, dateIncreasing: ✅
    57/232[1]: date: 1719007821000, dateIncreasing: ✅
    58/232[1]: date: 1719008444000, dateIncreasing: ✅
    59/232[1]: date: 1719008487000, dateIncreasing: ✅
    60/232[1]: date: 1719008653000, dateIncreasing: ✅
    61/232[1]: date: 1719021008000, dateIncreasing: ✅
    62/232[1]: date: 1719021069000, dateIncreasing: ✅
    63/232[1]: date: 1719021206000, dateIncreasing: ✅
    64/232[1]: date: 1719021288000, dateIncreasing: ✅
    65/232[1]: date: 1719000158000, dateIncreasing: ❌
    66/232[1]: date: 1719000158000, dateIncreasing: ✅
    67/232[1]: date: 1719000158000, dateIncreasing: ✅
    68/232[1]: date: 1718909485000, dateIncreasing: ❌
    
  5. wdanilo commented on Jun 25, 2024

    @wdanilo
    Author

    Moreover, as you can see in the above test, users.history.list can return records, that contain invalid message ids (the GMail API query error indicates so and I can reproduce it every time).

  6. ego commented on Oct 16, 2024

    @ego

    I also got the same case, and my sync process was broken due to these inconvenients.
    We periodically have missed this kind of messages.
    Any updates? Any API param for order?

  7. added
    needs more infoThis issue needs more information from the customer to proceed.
    on Feb 8, 2025
  8. sofisl commented on Feb 8, 2025

    @sofisl
    Contributor

    Hi there! Thanks for the writeup @wdanilo! As you mentioned this is likely an API issue. As this is mostly a generated library unfortunately we don't have much pull to help. I'll convert this to a discussion so people can follow along!

  9. locked and limited conversation to collaborators on Feb 8, 2025
  10. converted this issue into a discussion #3622 on Feb 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs more infoThis issue needs more information from the customer to proceed.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions