[Sugar-devel] Draft Proposal for Add AI to Chat activity

[Sujay R]

Yea sorry about that code snippet. I don't know why I added that, even I
don't understand it. I just wanted to let you know that I can figure out
the part of integrating the bot into the activity. That's why I added that
random snippet I've removed it.

I'm attaching my final proposal, that I'm submitting, as a reference.

Thank you for your assistance throughout the proposal review process

On Tue, Apr 2, 2024 at 12:04 AM Chihurumnaya Ibiam <ibiam at sugarlabs.org>
wrote:

> Please use the reply-all button to keep the conversation in the list.
>
> *"Please excuse me if I’m wrong, I’m not very familiar with the
> codebase.:"*
>
> You should remove this line, it serves no purpose, I also don't understand
> the code snippet
> you added and why.
>
> --
>
> Ibiam Chihurumnaya
> ibiam at sugarlabs.org
>
>
>
> On Sun, Mar 31, 2024 at 6:10 PM Sujay R <sujay1844 at gmail.com> wrote:
>
>> I've taken your advice and added more details about the implementation.
>> I'm sorry if sharing PDFs again and again is irritating. I'm sure you'll
>> let me know of a better method.
>>
>> I have asked you to review multiple versions of my draft proposal. I'm
>> very grateful for you take out your time and help me applying. Can you
>> please review it one last time? I'll submit once you give the green light.
>>
>> Thank you again for your guidance and support throughout this process
>>
>> On Fri, Mar 29, 2024 at 8:10 PM Chihurumnaya Ibiam <ibiam at sugarlabs.org>
>> wrote:
>>
>>> Your timeline is terse, it'll be great if you add some implementation
>>> detail to your proposal as there's
>>> scarcely any.
>>>
>>> --
>>>
>>> Ibiam Chihurumnaya
>>> ibiam at sugarlabs.org
>>>
>>>
>>>
>>> On Thu, Mar 28, 2024 at 7:54 PM Sujay R <sujay1844 at gmail.com> wrote:
>>>
>>>> Ok, the cloud sounds good.
>>>>
>>>> And thank you for patiently discussing the requirements with me and
>>>> answering my questions. Now, I think I can come up with a tentative
>>>> timeline.
>>>> I've attached v2 of my proposal with the timeline. Please let me know
>>>> if there are any more changes. If none, I'll submit it on the GSoC website.
>>>> The deadline is right around the corner
>>>>
>>>> On Wed, Mar 27, 2024 at 11:37 PM Chihurumnaya Ibiam <
>>>> ibiam at sugarlabs.org> wrote:
>>>>
>>>>> We were thinking of a cloud option, but we haven't decided yet.
>>>>>
>>>>> --
>>>>>
>>>>> Ibiam Chihurumnaya
>>>>> ibiam at sugarlabs.org
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 26, 2024 at 6:40 PM Sujay R <sujay1844 at gmail.com> wrote:
>>>>>
>>>>>> Thanks for explaining that. But it's still not clear to me where the
>>>>>> FOSS LLM should be run. Not on the devices running Sugar, so is cloud the
>>>>>> option you're looking for?
>>>>>>
>>>>>> On Tue, Mar 26, 2024 at 11:02 PM Chihurumnaya Ibiam <
>>>>>> ibiam at sugarlabs.org> wrote:
>>>>>>
>>>>>>> The plan was never really to run an LLM on Sugar as that'll
>>>>>>> drastically increase the size of the activity
>>>>>>> and Sugar itself as Chat is a fructose
>>>>>>> <https://wiki.sugarlabs.org/go/Development_Team/Release/Modules>
>>>>>>> activity.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Ibiam Chihurumnaya
>>>>>>> ibiam at sugarlabs.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 26, 2024 at 10:54 AM Sujay R <sujay1844 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sugar runs on a lot of devices including low end devices - 2GB ram
>>>>>>>>> - and we intend to keep it that way, the chat activity is
>>>>>>>>> typically used by more than one Sugar instance, the chatbot should
>>>>>>>>> also be able to run on just one instance.
>>>>>>>>
>>>>>>>>
>>>>>>>> Running LLMs on just 2GB ram is atleast a few years away. So the
>>>>>>>> bot has to be hosted on an API. Cloud is a good option, there is serverless
>>>>>>>> GPU inference and provisioned ones. One that I like is RunPod (serverless
>>>>>>>> pricing <https://www.runpod.io/serverless-gpu> and provisioned
>>>>>>>> pricing <https://www.runpod.io/gpu-instance/pricing>). Local
>>>>>>>> hosting is also an option, for a 7B model, a moderately new (4-5 year old)
>>>>>>>> GPU with 16GB VRAM. Running with lower RAM is possible but with excessive
>>>>>>>> quantisation (rounding off) at the cost of quality and speed.
>>>>>>>>
>>>>>>>>
>>>>>>>>> You can leverage the sugar-datastore if you need to store activity
>>>>>>>>> related data.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Storing the chat history is not an issue. Time complexity for
>>>>>>>> generation(inference) of a transfomer is O(n^2) where n is the number of
>>>>>>>> tokens. So we need to be mindful of how much history do we actually need
>>>>>>>>
>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20240402/7d40ff47/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sujay_R-Sugar_labs_GSoC_Chat_Activity_v4.pdf
Type: application/pdf
Size: 69201 bytes
Desc: not available
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20240402/7d40ff47/attachment-0001.pdf>