When the service does not have a public API, but there is a mailing list – Mikulski
Site Overlay

When the service does not have a public API, but there is a mailing list

By Mikulski 
DISCLAIMER:
I am not a programmer, but only an enthusiastic copypaster who shares what he was able to figure out and achieve efficiency. Therefore, it is possible that knowledgeable experts may find some points or formulations incorrect or ridiculous.
Submitting this material implies that you have some knowledge of the basics of node.js.
I'm using a node.js and pm2-manager.

Not so long ago, I added the display of the latest events on the stream for my 24/7 broadcast: follows, raids, etc. I hook events in different ways: part through the official Twitch and Telegram APIs, part through the Donation Alerts and Streamlabs web sockets. For Kick, for example, a more exotic technique: using the KickChatConnection library, I connect to my channel’s chat (without the ability to write there) and select messages by keywords. For example, when Botrix writes its reaction to an event, I cut this message to get a nickname:

if (data.includes("Thank you for the follow @") && jsonDataSub.sender.username === 'BotRix'){
console.log(jsonDataSub.content.match(/\w+$/)[0] + " is now following Kick")
   }
if (data.includes("Thank you for the raid @") && jsonDataSub.sender.username === 'BotRix'){ 
console.log(jsonDataSub.content.match(/\w+$/)[0] + " Kick Raid")
   }

It’s a little weird, but it works 🙂

But now it came to Boosty. Okay, folows, paid subscriptions and their renewals are great coming through the Donation Alerts socket. The only thing missing is the donations sent via the Boosty page.
Of course, there are craftsmen who are subject to reverse engineering techniques in order to implement a connection directly to the website API, but, of course, I am not one of them. In general, having failed to find any example of using the unofficial Boosty API that would work for me, I went to bed.
I’m lying and thinking, but how do I get these donation events? And then I realize that their Telegram bot sends notifications, and they also come in the mailing list. The dream disappeared, and I went to Google if there was any way to parse private messages coming from the bot in Telegram. With Telegram, this would be the easiest and most convenient, but nothing on this topic could be found. This is probably not possible for security reasons. But to implement such an approach via mail via the IMAP protocol turned out to be quite real.

Mailbox

Not being sure how safe it is to parse emails to a remote server, I decided that it was worth having a separate mailbox for this case. Which does not receive messages with “sensitive” information. The benefit of Boosty, namely, for notifications, allows you to link any mail that is not used for logging into your account. But it is possible that this is an unnecessary precaution, since the connection takes place through an encrypted gateway on port 993 with tls.
I opened such a mailbox on Yandex.
To enable the transfer of emails over the IMAP protocol for it, go to the mail settings (gear icon), then:
All settings -> Email clients -> check the box "User a mail client to retrieve your Yandex mail: From the imap.yandex.com server via IMAP".

According to the requirements of the service, you need to generate an application password. To do this, go to the “Security” tab of the Yandex account (Account Management -> Security), go down the page and go to the “App Passwords” tab -> Create an app password -> Email address.
Here you will be asked to set a name for the password (not the password, but its name, for example, imap-boosty), after which a password will be generated. It is shown only once, before closing the tab, so you need to copy and save it.

node-imap

To connect to the mailbox, I use the imap library, and for parsing the text of messages, mailparser. The imap documentation is not very noob-friendly, but you can figure it out: there are quite a lot of different options for manipulating mail.

npm i imap
npm i mailparser

With the default settings and those given in the documentation examples, the connection socket is terminated too often. The script, of course, will still automatically restore the connection, but it’s still worth tweaking the keep alive settings to reduce the number of discounts. It looks something like this to me:

var fs = require('fs'), fileStream;
const Imap = require("imap");
const { inspect } = require("util");
const simpleParser = require('mailparser').simpleParser;

const imapServer = new Imap({
  user: "mailbox_name@yandex.com",
  password: "the_generated_app_password",
  host: "imap.yandex.com",
  port: 993,
  tls: true,
  keepalive: {
            interval: 5000,
            idleInterval: 120000,
            forceNoop: true
        }
});

To make sure that this works, you need to create a connection, open the inbox and check if there are unread messages (send yourself an email and make sure that it is marked unread):

imapServer.once('ready', function() {
imapServer.openBox('INBOX', false, function (err, box) {
if (err) throw err;
   
imapServer.search(['UNSEEN'], async (err, results) => {
if (err) throw err;
if (results && results.length > 0) {
console.log("There are unread messages!")
    }
    });
  });
});

imapServer.once("error", (err) => {
  console.log(err);
});
imapServer.once("end", () => {
  console.log("Connection ended");
});
imapServer.connect(console.log('imap connected'));

This check is performed only once (the ‘ready’ tag) on the first connection and is needed to open the inbox.
To receive a response to new unread messages, the ‘mail’ tag is used:

imapServer.on('mail', function() {
imapServer.search(['UNSEEN'], async (err, results) => {         
if (err) throw err;
if (results && results.length > 0) {
console.log("New message!")
    }
  });
})

Okay, now, in order to fish out emails with the right subject from the right recipient and mark them as read, you need to rewrite the search terms (for example, a donation notification on Boosty):

imapServer.on('mail', function() {
imapServer.search(['UNSEEN', ['FROM', "noreply@boosty.to"], ['HEADER', 'SUBJECT' ,"You have a new donation"]], async (err, results) => {    
if (err) throw err;
if (results && results.length > 0) {
var f = imapServer.fetch(results, {
bodies: '',
markSeen: true      
});
console.log('Donation on Boosty!')
}
});
});

The whole thing:

var fs = require('fs'), fileStream;
const Imap = require("imap");
const { inspect } = require("util");
const simpleParser = require('mailparser').simpleParser;

const imapServer = new Imap({
  user: "mailbox_name@yandex.com",
  password: "the_generated_app_password",
  host: "imap.yandex.com",
  port: 993,
  tls: true,
  keepalive: {
            interval: 5000,
            idleInterval: 120000,
            forceNoop: true
        }
});

imapServer.once('ready', function() {
imapServer.openBox('INBOX', false, function (err, box) {
if (err) throw err;
   
imapServer.search(['UNSEEN'], async (err, results) => {
if (err) throw err;
if (results && results.length > 0) {
console.log("There are unread messages!")
    }
    });
  });
});

imapServer.on('mail', function() {
imapServer.search(['UNSEEN', ['FROM', "noreply@boosty.to"], ['HEADER', 'SUBJECT' ,"You have a new donation"]], async (err, results) => {    
if (err) throw err;
if (results && results.length > 0) {
var f = imapServer.fetch(results, {
bodies: '',
markSeen: true      
});
console.log('Donation on Boosty!')
};
});
});

imapServer.once("error", (err) => {
  console.log(err);
});
imapServer.once("end", () => {
  console.log("Connection ended");
});
imapServer.connect(console.log('imap connected'));

In general, this is already enough to get a trigger for further manipulations.
However, you can read the contents of the email and try to extract the necessary information from it: the user and the amount.
I did it through the split method. It is clear that in this case, everything will depend on the layout of the letter. If it changes over time, then the “clipping” of the necessary data will have to be redone. However, I think that from the very beginning of this venture it is clear that this whole workaround is a compromise and an extreme option.
And with the subject “You have a new donation”, donations come for different purposes: just donation, donation for the goal, donation under the post. And they all have different layouts, therefore, the split parameters will be different for all of them.
I’ll show you an example of donating to a target:

imapServer.search(['UNSEEN', ['FROM', "noreply@boosty.to"], ['HEADER', 'SUBJECT' ,"You have a new donation"]], async (err, results) => {
if (err) throw err;
if (results && results.length > 0) {
var f = imapServer.fetch(results, {
bodies: '',
markSeen: true
})
f.on('message', function(msg, seqno) {
msg.on('body', function(stream, info) {  
simpleParser(stream, (err, mail) => {
if (mail.text.includes('You have a new donation for the goal')) {
let username = mail.text.split(" ", 8)[7].slice(5);
let amount = mail.text.split(" ", 18)[17] + " RUB"
console.log(`${username} donated ${amount} on Boosty!`);
      };
    });
   });     
  });
 };
});

Since parsing the message body takes some time (it varies from case to case), it makes sense to add a timeout of several seconds before performing further actions with the received data. For example, to write to a file:

imapServer.search(['UNSEEN', ['FROM', "noreply@boosty.to"], ['HEADER', 'SUBJECT' ,"You have a new donation"]], async (err, results) => {
if (err) throw err;
if (results && results.length > 0) {
var f = imapServer.fetch(results, {
bodies: '',
markSeen: true
})
f.on('message', function(msg, seqno) {
msg.on('body', function(stream, info) {  
simpleParser(stream, (err, mail) => {
if (mail.text.includes('You have a new donation for the goal')) {
let username = mail.text.split(" ", 8)[7].slice(5);
let amount = mail.text.split(" ", 18)[17] + " RUB"
console.log(`${username} donated ${amount} on Boosty!`);

setTimeout(wait, 3000)
fs.writeFile('/path/to/file/', `${username} donated ${amount} on Boosty!`, function(err) {if (err) return console.log(err)});       
  
function wait() {
process.exit(0);     
}
      };
    });
   });     
  });
 };
});

Even after performing this operation, I added process.exit(0) so that the script stops working and automatically restarts by pm2-manager. Because at an early stage of testing, I noticed that the script stopped responding to emails with the same subject after a certain number of attempts. Therefore, I decided to hedge my bets this way.

In the end, this whole workaround looks something like this (I haven’t tested this code, if anything, don’t blame me):

var fs = require('fs'), fileStream;
const Imap = require("imap");
const { inspect } = require("util");
const simpleParser = require('mailparser').simpleParser;

const imapServer = new Imap({
  user: "mailbox_name@yandex.com",
  password: "the_generated_app_password",
  host: "imap.yandex.com",
  port: 993,
  tls: true,
  keepalive: {
            interval: 5000,
            idleInterval: 120000,
            forceNoop: true
        }
});

imapServer.once('ready', function() {
imapServer.openBox('INBOX', false, function (err, box) {
if (err) throw err;
   
imapServer.search(['UNSEEN'], async (err, results) => {
if (err) throw err;
if (results && results.length > 0) {
console.log("There are unread messages!")
    }
    });
  });
});

imapServer.search(['UNSEEN', ['FROM', "noreply@boosty.to"], ['HEADER', 'SUBJECT' ,"You have a new donation"]], async (err, results) => {
if (err) throw err;
if (results && results.length > 0) {
var f = imapServer.fetch(results, {
bodies: '',
markSeen: true
})
f.on('message', function(msg, seqno) {
msg.on('body', function(stream, info) {  
simpleParser(stream, (err, mail) => {
if (mail.text.includes('You have a new donation for the goal')) {
let username = mail.text.split(" ", 8)[7].slice(5);
let amount = mail.text.split(" ", 18)[17] + " RUB"
console.log(`${username} donated ${amount} on Boosty!`);

setTimeout(wait, 3000)
fs.writeFile('/путь/к/файлу/', `${username} donated ${amount} on Boosty!`, function(err) {if (err) return console.log(err)});       
  
function wait() {
process.exit(0);     
}
      };
    });
   });     
  });
 };
});     


imapServer.once("error", (err) => {
  console.log(err);
});
imapServer.once("end", () => {
  console.log("Connection ended");
});
imapServer.connect(console.log('imap connected'));

Here is such a strange solution, based on my skills, I was able to come up with and implement to complete the task. I don’t think this will be really useful to anyone, but it may inspire or amuse 😉

0 comments
Inline Feedbacks
View all comments