Generative AI Study Group_15thSesssion_20240116

Generative AI Study Group

Agenda

Agenda

Introduction • 今日のパネラー紹介 • 勉強会貢献メンバー

Introduction How make AI do it is all you need!!

Introduction Yes, AI can!!

Introduction • 公式X • https://twitter.com/kunihirosks

• https://twitter.com/kunihirosks

Introduction • 前回の振り返り • Web更新 • https://www.ai-tech-c.jp/generative-ai-study-group-gasg/ • アンケート結果 • 今回のアンケート

• https://www.ai-tech-c.jp/generative-ai-study-group-gasg/

Introduction •進捗 • テーマとゲスト急募!! • 新企画 • セキュリティ • アルファブロガー hoge様

Introduction • 参加者名簿 • 個別にご連絡をさせていただくため活用させていただく場合があ りますので、是非。 • Form • https://forms.gle/eRmeGtfkQMkDrs7K9

• https://forms.gle/eRmeGtfkQMkDrs7K9

Introduction •AITeCに是非ご入会をご検討ください •驚きの安さ!! •なんと年間10万円ポッキリ!! •安い! 楽しい! 凄い! AITeC!!

Introduction • AITeC会員限定 • Mattermost • • 情報 / 企画 / ビジョン • 厳選とれたて新鮮Info from bookmarks • 僕が一番ChatGPTをうまくつかえるんだぁー!!大会 • ChatGPT相談室 • 疑似人格 • AIを使った社会実験 • Generative Agents • 新AIパラダイム時代の生き方論 • Deparure of Computable Fundamentalism ABCI利用 • スーパーエンジニアが親切にサポート (予定

Introduction • AITeCオープンイベント o【ABCIでLLMを動かそう！セミナー 第2回】 機械学 習・ディープラーニングとは - 人工知能技術コンソー シアム | Doorkeeper https://aitconsortium.doorkeeper.jp/events/168119

• https://aitconsortium.doorkeeper.jp/events/168119

Agenda

Theme • Title • NISTEPによる生成AIとDXのすがた • Presenter • 盛岡広域振興局 佐藤 清忠氏 • Contents • https://drive.google.com/file/d/1kU6PVDymrcyABr59ag90Nb4VpywZ0WB/view?usp=sharing

• https://drive.google.com/file/d/1kU6PVDymrc-yABr59ag90Nb4VpywZ0WB/view?usp=sharing

Theme • Title • 2024年生成系AIの展望と考察 ▪ Mainly NLP topics • Presenter • 杉山邦洋 @Generative Ai Study Group .host • Contents • Tech trend, Use case, Issue

Theme •「マルチモーダルAI」「小規模言語モデ ル」2024年の生成AI重要トレンド（Forbes JAPAN） - Yahoo!ニュース https://news.yahoo.co.jp/articles/633019417533 9a101c2f6e8ad36b0c4109943939?page=1

• https://news.yahoo.co.jp/articles/6330194175339a101c2f6e8ad36b0c4109943939?page=1

Tech trend

Tech trend •Small model •Beyond Transformer •Related tech

Tech trend • Small model • Open LLM Leaderboard - a Hugging Face Space by HuggingFaceH4 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard • Points ▪ コンピューティング量 ▪ 実行メモリ量 ▪ 推論速度 ▪ 特化型カスタマイゼーション

• https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Tech trend • Small model • Pickup ▪ TinyLlama • jzhang38/TinyLlama:The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens. https://github.com/jzhang38/TinyLlama • 【軽量かつ高速なLLM】TinyLlamaについてまとめてみた #LLM - Qiita https://qiita.com/sergicalsix/items/7cd7665ab90b9f3b343c • LLAMAと完全互換のアーキテクチャおよびトークンナイザー • 1.1Bパラメータモデルは4bit量子化でおよそ550MB RAM上で動作

• https://github.com/jzhang38/TinyLlama
• https://qiita.com/sergicalsix/items/7cd7665ab90b9f3b343c

Tech trend • Small model • Pickup ▪ Phi-2 • Phi-2: The surprising power of small language models - Microsoft Research https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-powerof-small-language-models/ • [2306.11644] Textbooks Are All You Need https://arxiv.org/abs/2306.11644 • 2.7Bパラメータモデル • 高品質な学習データセットでモデル品質を確保

• https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
• https://arxiv.org/abs/2306.11644

Tech trend • Small model • Pickup ▪ Orca 2 • Orca - Microsoft Research https://www.microsoft.com/en-us/research/project/orca/ • Microsoft's Orca 2 LLM Outperforms Models That Are 10x Larger https://www.infoq.com/news/2023/12/microsoft-orca-2-llm/ • 7B, 13Bパラメータモデル • LLAMA-2のFinetunedモデル • Reasoningが含まれる合成データセットで訓練

• https://www.microsoft.com/en-us/research/project/orca/
• https://www.infoq.com/news/2023/12/microsoft-orca-2-llm/

Tech trend • Small model • Pickup ▪ DeciLM • Deci/DeciLM-7B · Hugging Face https://huggingface.co/Deci/DeciLM-7B • [2305.13245] GQA:Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints https://arxiv.org/abs/2305.13245 • What is Grouped Query Attention (GQA)? — Klu https://klu.ai/glossary/grouped-queryattention • GQAの仕組みを採用し軽量で高性能 • GQAは例えばLLaMA2 70Bでも使われている

• https://huggingface.co/Deci/DeciLM-7B
• https://arxiv.org/abs/2305.13245
• https://klu.ai/glossary/grouped-query-attention

Tech trend 二次的複雑性 (Quadratic complexity O(n^2)) http://tinyurl.com/yqs8pwec • Beyond Transformer • Points ▪ Transformerの課題解決の試み • 推論速度 • 実行メモリ量 • シーケンス長 • コンピューティング量

• http://tinyurl.com/yqs8pwec

Tech trend • Beyond Transformer • Pickup ▪ MoE (Mixture of Experts) Reference: http://tinyurl.com/ylxsvomj

• http://tinyurl.com/ylxsvomj​

Tech trend • Beyond Transformer • Pickup ▪ MoE (Mixture of Experts) • 論文解説 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (MoE) - ディープラーニングブログ https://deeplearning.hatenablog.com/entry/moe • Mixture of expertsの簡単な実装をしてみる https://zenn.dev/if001/articles/40917524959913 • MixtralSparseMoeBlockを読む https://zenn.dev/if001/articles/fcea9fe9f1bdb1

• https://deeplearning.hatenablog.com/entry/moe
• https://zenn.dev/if001/articles/40917524959913
• https://zenn.dev/if001/articles/fcea9fe9f1bdb1

Tech trend • Beyond Transformer • Pickup ▪ RWKV (Reinventing RNNs for the Transformer Era) • RWKVについて解説 | AGIRobots Blog https://developers.agirobots.com/jp/rwkv/ • RWKVを論文と実装から読み解く https://zenn.dev/jow/articles/f66d6403b9a509 • RNNでTransformer並みの性能を実現するRWKVがやばい https://zenn.dev/hikettei/articles/5d6c1318998411 • これは衝撃!1.5Bで超高性能LLM!RWKV-5-World-v2｜shi3z https://note.com/shi3zblog/n/nfc8dd1abf494

• https://developers.agirobots.com/jp/rwkv/
• https://zenn.dev/jow/articles/f66d6403b9a509
• https://zenn.dev/hikettei/articles/5d6c1318998411
• https://note.com/shi3zblog/n/nfc8dd1abf494

Tech trend • Beyond Transformer • Pickup ▪ Mamba • state-spaces/mamba https://github.com/state-spaces/mamba • [2312.00752] Mamba: Linear-Time Sequence Modeling with Selective State Spaces https://arxiv.org/abs/2312.00752 • Mamba: Redefining Sequence Modeling and Outforming Transformers Architecture - Unite.AI https://www.unite.ai/mambaredefining-sequence-modeling-and-outforming-transformers-architecture/ • Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Arxiv Dives | by Oxen | Dec, 2023 | Medium https://medium.com/@oxenai/mamba-linear-time-sequence-modeling-with-selective-state-spaces-arxiv-dives-cf96518d7ec4 ▪ StripedHyena • Architectures for longer sequences and efficient inference: StripedHyena | hessian.AI https://hessian.ai/architectures-for-longersequences-and-efficient-inference-stripedhyena/ • [2302.10866] Hyena Hierarchy:Towards Larger Convolutional Language Models https://arxiv.org/abs/2302.10866

• https://github.com/state-spaces/mamba
• https://arxiv.org/abs/2312.00752
• https://www.unite.ai/mamba-redefining-sequence-modeling-and-outforming-transformers-architecture/
• https://medium.com/@oxenai/mamba-linear-time-sequence-modeling-with-selective-state-spaces-arxiv-dives-cf96518d7ec4
• https://hessian.ai/architectures-for-longer-sequences-and-efficient-inference-stripedhyena/
• https://arxiv.org/abs/2302.10866

Tech trend • Beyond Transformer • Pickup ▪ MoE-Mamba • [2401.04081] MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts https://arxiv.org/abs/2401.04081 ▪ Monarch Mixer • [2310.12109] Monarch Mixer: A Simple Sub-Quadratic GEMMBased Architecture https://arxiv.org/abs/2310.12109

• https://arxiv.org/abs/2401.04081
• https://arxiv.org/abs/2310.12109

Tech trend •Related tech •RAG (Retrieval-Augmented Generation) •Agent •Synthetic data (合成データ) •Distributed

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪第6回で取り上げました ▪進化するRAGアーキテクチャ ▪[2312.10997] Retrieval-Augmented Generation for Large Language Models:A Survey https://arxiv.org/abs/2312.10997

• https://arxiv.org/abs/2312.10997

Reference: https://arxiv.org/pdf/2312.10997.pdf Figure 6: RAG compared with other model optimization methods Theme • Tech Trend • Related tech ▪ RAG (Retrieval-Augmented Generation)

• https://arxiv.org/pdf/2312.10997.pdf

Reference: https://arxiv.org/pdf/2312.10997.pdf Figure 2: A representative instance of the RAG process applied to question answering Theme • Tech Trend • Related tech ▪ RAG (Retrieval-Augmented Generation)

• https://arxiv.org/pdf/2312.10997.pdf

Reference: https://arxiv.org/pdf/2312.10997.pdf Figure 3: Comparison between the three paradigms of RAG Theme • Tech Trend • Related tech ▪ RAG (Retrieval-Augmented Generation)

• https://arxiv.org/pdf/2312.10997.pdf

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪Advanced • Optimizing data indexing • Pre retrieval process • Post retrieval process

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪ Modular • 多様な機能モジュール • ニーズに適したPipeline

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪ A Cheat Sheet and Some Recipes For Building Advanced RAG | by Andrei | Jan, 2024 | LlamaIndex Blog https://blog.llamaindex.ai/a-cheat-sheet-and-somerecipes-for-building-advanced-rag-803a9d94c41b

• https://blog.llamaindex.ai/a-cheat-sheet-and-some-recipes-for-building-advanced-rag-803a9d94c41b

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪ Scaling context window ▪ Robustness • Hallucination ▪ Hybrid (RAG+FT) ▪ Expanding LLM role ▪ Scaling law • Embedding model ▪ Production ready • 精度, 再現性, セキュリティ(アクセスコントロール) ▪ Multi modal • Image, Audio and video, Code

Tech trend • Related tech • Agent ▪ 第11回で取り上げました Reference: https://medium.com/scisharp/understand-the-llm-agent-orchestration-043ebfaead1f

• https://medium.com/scisharp/understand-the-llm-agent-orchestration-043ebfaead1f

Tech trend • Related tech • Agent ▪ 2024 AI Agent ▪ https://e2b.dev/blog/ai-agents-in-2024 ▪ 評価フレームワーク • THUDM/AgentBench:A Comprehensive Benchmark to Evaluate LLMs as Agents https://github.com/THUDM/AgentBench • AutoGPT/benchmark at master · Significant-Gravitas/AutoGPT https://github.com/Significant-Gravitas/AutoGPT/tree/master/benchmark • Benchmarking Agent Tool Use https://blog.langchain.dev/benchmarking-agent-tool-use

• https://e2b.dev/blog/ai-agents-in-2024
• https://github.com/THUDM/AgentBench
• https://github.com/Significant-Gravitas/AutoGPT/tree/master/benchmark
• https://blog.langchain.dev/benchmarking-agent-tool-use

Tech trend • Related tech • 合成データ ▪ Synthetic data:Anthropic’s CAI, scaling, OpenAI’s Superalignment, tips, and open-source examples https://www.interconnects.ai/p/llm-synthetic-data ▪ [2305.15041] Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science https://arxiv.org/abs/2305.15041 ▪ [2310.07849] Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations https://arxiv.org/abs/2310.07849 ▪ [2401.00368] Improving Text Embeddings with Large Language Models https://arxiv.org/abs/2401.00368 ▪ [2312.17742] Learning Vision from Models Rivals Learning Vision from Data https://arxiv.org/abs/2312.17742

• https://www.interconnects.ai/p/llm-synthetic-data
• https://arxiv.org/abs/2305.15041
• https://arxiv.org/abs/2310.07849
• https://arxiv.org/abs/2401.00368
• https://arxiv.org/abs/2312.17742

Tech trend • Related tech • Distributed ▪ Petals • Petals – Run LLMs at home, BitTorrent-style https://petals.dev/ • bigscience-workshop/petals: Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading https://github.com/bigscience-workshop/petals • [2209.01188] Petals: Collaborative Inference and Fine-tuning of Large Models https://arxiv.org/abs/2209.01188 • [2312.08361] Distributed Inference and Fine-tuning of Large Language Models Over The Internet https://arxiv.org/abs/2312.08361 ▪ AI Horde • AI Horde https://stablehorde.net/ • Haidra-Org/AI-Horde: A crowdsourced distributed cluster for AI art and text generation https://github.com/Haidra-Org/AI-Horde?tab=readme-ov-file

• https://petals.dev/
• https://github.com/bigscience-workshop/petals
• https://arxiv.org/abs/2209.01188
• https://arxiv.org/abs/2312.08361
• https://stablehorde.net/
• https://github.com/Haidra-Org/AI-Horde?tab=readme-ov-file

Use case

Use case • 企業における生成AIの未来：ChatGPTを越えてその先へ | ガートナー https://www.gartner.co.jp/ja/articles/beyond-chatgptthe-future-of-generative-ai-for-enterprises • Top 100+ Generative AI Applications / Use Cases in 2024 https://research.aimultiple.com/generative-ai-applications/ • 2024 AI Predictions | NVIDIA Blog https://blogs.nvidia.com/blog/2024-ai-predictions/

• https://www.gartner.co.jp/ja/articles/beyond-chatgpt-the-future-of-generative-ai-for-enterprises
• https://research.aimultiple.com/generative-ai-applications/
• https://blogs.nvidia.com/blog/2024-ai-predictions/

Use case • Device ▪ Order Ai Pin Now https://hu.ma.ne/ ▪ rabbit — home https://www.rabbit.tech/ ▪ Avi Schiffmann’s Tab AI necklace raised $1.9 million to replace God https://www.fastcompany.com/91007630/avi-schiffmanns-tab-ai-necklace-hasraised-1-9-million-to-replace-god ▪ OpenAI、「AI版iPhone」をアップル元デザイナーと開発か 孫正義に 出資打診 | Forbes JAPAN 公式サイト（フォーブス ジャパン） https://forbesjapan.com/articles/detail/66348

• https://hu.ma.ne/
• https://www.rabbit.tech/
• https://www.fastcompany.com/91007630/avi-schiffmanns-tab-ai-necklace-has-raised-1-9-million-to-replace-god
• https://forbesjapan.com/articles/detail/66348

Issue

Issue •Security •Data contamination •Socialization

Issue • Security • Overview ▪ 8 Generative AI Security Risks That You Should Know https://www.globalsign.com/en/blog/8-generative-ai-security-risks ▪ Safety and security risks of generative artificial intelligence to 2025 (Annex B) - GOV.UK https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/safety-and-security-risks-of-generativeartificial-intelligence-to-2025-annex-b • Prompt hack • GPTs のプロンプトリーキング対策｜ぬこぬこ https://note.com/schroneko/n/n6d6c2e645119 • Prompt Hacking | Learn Prompting:Your Guide to Communicating with AI https://learnprompting.org/docs/category/-prompt-hacking • Solution ▪ Introducing Purple Llama for Safe and Responsible AI Development | Meta https://about.fb.com/news/2023/12/purple-llama-saferesponsible-ai-development/ ▪ New generative AI-powered SaaS security expert from AppOmni | VentureBeat https://venturebeat.com/security/new-generative-aipowered-saas-security-expert-from-appomni

• https://www.globalsign.com/en/blog/8-generative-ai-security-risks
• https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/safety-and-security-risks-of-generative-artificial-intelligence-to-2025-annex-b
• https://note.com/schroneko/n/n6d6c2e645119
• https://learnprompting.org/docs/category/-prompt-hacking
• https://about.fb.com/news/2023/12/purple-llama-safe-responsible-ai-development/
• https://venturebeat.com/security/new-generative-ai-powered-saas-security-expert-from-appomni

Issue • Data contamination • Why data contamination is a big issue for LLMs - TechTalks https://bdtechtalks.com/2023/07/17/llm-data-contamination/ • [2312.16337] Task Contamination: Language Models May Not Be Few-Shot Anymore https://arxiv.org/abs/2312.16337 • Socialization • Sotopia https://www.sotopia.world/ • [2310.11667] SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents https://arxiv.org/abs/2310.11667

• https://bdtechtalks.com/2023/07/17/llm-data-contamination/
• https://arxiv.org/abs/2312.16337
• https://www.sotopia.world/
• https://arxiv.org/abs/2310.11667

Agenda

Study contents • Coming Sooooooooooon!!

Agenda

News • みんなで生成系AIブックマーク • 新企画です!! 皆様の気になった生成系AIの情報をその場で募集して ディスカッションします。 • 募集はこちら https://forms.gle/kKVsb9jDwMTm2Rkk6 から事前投稿も受 け付けています。 • リスト • https://docs.google.com/spreadsheets/d/1bjJfAhfEvNMSuxAszCiSo0qdE89s p86ooAyBdRVuDO0/edit?usp=sharing

• https://forms.gle/kKVsb9jDwMTm2Rkk6
• https://docs.google.com/spreadsheets/d/1bjJfAhfEvNMSuxAszCiSo0qdE89sp86ooAyBdRVuDO0/edit?usp=sharing

Agenda

Next • アンケートの結果とコメント • 次回 • みんなで生成系AIブックマーク" 継続 • 分析してみる？ • Theme引き続き募集 • Study contents • セキュリティ • Fine tuning • ChatGPT Plugins • オープンソース各種 • Langchain,LlamaIndex, 他 • Business向けコンテンツ • New era personal computer

EOF https://www.linkedin.com/in/kunihiro-sugiyama-49b0372a/ https://www.ai-tech-c.jp/generative-ai-study-group-gasg/

• https://www.linkedin.com/in/kunihiro-sugiyama-49b0372a/
• https://www.ai-tech-c.jp/generative-ai-study-group-gasg/