ShareGPT 数据集格式

文章目录一、格式概览常见数据集格式对比Alpaca ≠ 单轮ShareGPT ≠ 多轮重要澄清二、字段详解from 字段的取值三、设计哲学四、加载与使用HuggingFace datasets下载数据集查看数据样例五、常见变体与注意事项六、适用场景与局限性七、从 ShareGPT 格式到训练输入八、总结一、格式概览ShareGPT 格式是一种对话式微调数据的通用结构核心思想是将多轮人机对话组织为conversations列表每个条目代表一轮发言明确标注发言者角色from和内容value。{id:L1VldBF_0,conversations:[{from:human,value:你好能帮我写一篇博客吗},{from:gpt,value:当然可以请告诉我主题。},{from:human,value:关于AI数据集格式的。},{from:gpt,value:很好的主题我们开始吧……}]}常见数据集格式对比除 ShareGPT 外主流格式还有以下几种格式核心字段适用场景Alpacainstruction,input,output指令微调文本生成、摘要、翻译等单轮任务ShareGPTconversations列表fromvalue多轮对话系统训练Messages (OpenAI)messages列表rolecontent标准 Chat API 格式DPO/ORPOchosen,rejected偏好对齐训练Alpaca ≠ 单轮ShareGPT ≠ 多轮重要澄清这是一个常见误解。Alpaca 格式同样支持多轮对话通过history字段存储历史对话。单轮时省略该字段即可。ShareGPT 格式同样支持单轮对话conversations列表只有一轮问答即为单轮。两者的本质差异不在于轮次而在于数据组织哲学Alpaca以指令-输入-输出三元组为核心强调任务定义的清晰性。即便多轮也是把历史对话作为附加上下文。ShareGPT以对话流为核心强调角色交互的连续性天然模拟真人聊天场景。一句话总结Alpaca 更适合做任务ShareGPT 更适合聊天。轮次只是表现形式不是格式的边界。二、字段详解字段类型必选说明idstring推荐样本唯一标识便于追溯和去重conversationslist[dict]必选对话轮次序列按时间顺序排列fromstring必选发言者角色通常为human或gpt也可见system、assistant等变体valuestring必选实际发言内容纯文本或含Markdownmarkdownbool/null可选指示value是否包含Markdown格式老旧字段逐渐弃用textstring/null可选替代value的遗留字段当前实际以value为主关键点conversations数组本身不显式区分系统提示、用户输入、助手回复而是通过from取值区分角色。训练时需按既定角色映射如human→ user,gpt→ assistant构造messages。from字段的取值from字段的取值不限于human和gpt。随着模型能力增强和应用场景复杂化已扩展出多种角色。常见取值包括角色说明system系统提示设定模型行为或人格human/user人类用户输入gpt/assistantAI 助手的回复function_call模型请求调用某个外部函数/工具observation函数调用后返回的观测结果tool工具/函数的描述信息function_call和observation的引入使 ShareGPT 格式能够支持函数调用Function Calling场景[{from:human,value:这件200块的裙子打8折后多少钱},{from:function_call,value:{\name\: \calculate_discount\, \arguments\: {\original_price\: 200, \discount_percentage\: 20}}},{from:observation,value:{\discounted_price\: 160}},{from:gpt,value:打8折后是160元。}]注意不同框架对角色名称的映射可能不同。例如转换为 Llama2 格式时human→user、gpt→assistant、system→system。部分实现也允许自定义角色标签如field_human、field_model等。三、设计哲学扁平化所有轮次平铺在数组中无需嵌套history或context简单直观易于拼接。角色明确from字段直接标明发言者无需额外推断兼容多种对话范式闲聊、问答、指令遵循等。多用途既可用于监督微调SFT也可经转换用于RLHF奖励模型、PPO或DPO。无固定模板数据集中不预设 chat template如|user|、|assistant|由下游加载时动态应用保持数据纯净性。四、加载与使用HuggingFacedatasets为了快速验证目前以ShareGPT_V3_unfiltered_cleaned_small_9k数据集为例如需大规模训练可换用完整的anon/ShareGPT_Vicuna_unfiltered或Aeala/ShareGPT_Vicuna_unfiltered。https://huggingface.co/datasets/mychen76/ShareGPT_V3_unfiltered_cleaned_small_9k?librarydatasets下载数据集使用datasets.load_dataset直接加载返回DatasetDict包含train/test/valid分割。fromdatasetsimportload_dataset dsload_dataset(mychen76/ShareGPT_V3_unfiltered_cleaned_small_9k)fromdatasetsimportload_dataset......dsload_dataset(mychen76/ShareGPT_V3_unfiltered_cleaned_small_9k)...README.md:100%|██████████████████████████████████████████████████████████████████████████████████████████████|844/844[00:0000:00,516kB/s]data/train-00000-of-00001-008084c40c6708(…):100%|███████████████████████████████████████████████████████|25.3M/25.3M[00:0400:00,5.53MB/s]data/test-00000-of-00001-c70edcc42ea72ad(…):100%|███████████████████████████████████████████████████████|2.74M/2.74M[00:0000:00,3.82MB/s]data/valid-00000-of-00001-24a3c3c079cd49(…):100%|██████████████████████████████████████████████████████████|297k/297k[00:0200:00,136kB/s]Generating train split:100%|███████████████████████████████████████████████████████████████████|8473/8473[00:0000:00,64187.55examples/s]Generating test split:100%|██████████████████████████████████████████████████████████████████████|942/942[00:0000:00,67324.99examples/s]Generating valid split:100%|███████████████████████████████████████████████████████████████████████|95/95[00:0000:00,34507.57examples/s]ds DatasetDict({train:Dataset({features:[id,conversations],num_rows:8473})test:Dataset({features:[id,conversations],num_rows:942})valid:Dataset({features:[id,conversations],num_rows:95})})ds[train]Dataset({features:[id,conversations],num_rows:8473})ds[train][:10]访问训练集样本sampleds[train][0]print(sample[id])print(len(sample[conversations]))print(sample[conversations][0][from],sample[conversations][0][value][:50])查看数据样例ds[train][:2]{id:[L1VldBF_0,MtrF4Lm_89],conversations:[[],[{from:gpt,markdown:None,text:None,value:In this piece of writing, Justin talks about the It sells itself mentality in business and how it is complete nonsense. He gives an example of a pizza shop that has a steady clientele from word of mouth advertising but emphasizes that this is a rare instance. He also talks about a record shop owner who refuses to do any advertising, thinking it is a waste of money, but Justin believes he is aiming his message in front of the wrong audience. Justin stresses the importance of continuously doing what is making money until it stops making money, and that not everyone knows about a business, so advertising is crucial.\n\nJustin also shares a story about a martial arts studio owner who says he doesn\t care about money and marketing, but Justin believes marketing is essential, even if the business is just breaking even. Justin thinks it\s funny that the Sensi talked about not wanting to do marketing when the conversation had everything to do with it. Throughout this piece of writing, Justin\s writing style is conversational and relaxed, with occasional tangents that make his points more relatable to the reader. He also uses personal anecdotes to reinforce his arguments and add a personal touch to the piece.},{from:human,markdown:None,text:None,value:for Justins writing we left off on page 128},{from:gpt,markdown:None,text:None,value:Im sorry, but I dont have access to any specific writing by Justin. Could you please provide me with more context or information so I can assist you better?},{from:human,markdown:None,text:None,value:i am noting we left off on page 128 of Justin McDonalds blogs},{from:gpt,markdown:None,text:None,value:Thank you for letting me know. Please provide the link to Justin McDonalds blog so that I can access it and continue from where we left off.},{from:human,markdown:None,text:None,value:twitter tags for: Curiosity Killed the Cat, but Saved the\xa0Marketer blog},{from:gpt,markdown:None,text:None,value:#CuriosityKilledTheCat #SavedTheMarketer #MarketingBlog #DigitalMarketing #ContentMarketing #SocialMediaMarketing #Curiosity #MarketingTips #MarketingStrategy #MarketingAdvice #MarketingInsights},{from:human,markdown:None,text:None,value:twitter tags for: 5 Ways to Legally Boost Your Productivity at Work and make more than your boss makes\nAre you struggling to stay productive at work?\nDistracted or find yourself procrastinating?\nIt can be challenging to get things done when youre not in the right mindset. \nBut dont worry my dear entrepreneurs, there are a few things you can do to boost your productivity and stay focused throughout the day.\nHere are five tips to rock roll and help you stay on track and get more done:\n1. Set clear goals - Start each day with a clear plan of what you want to accomplish. Write down your goals and prioritize them based on importance. But always have the next goal(s) in mind as you cross that finish line.\n2. Eliminate distractions - Minimize interruptions by turning off notifications on your phone and email. Find a quiet place to work if possible. (Bonus: if you can find peace in chaos, the youll have an easier time with this. Practice reading or writing at busy times at bars and restaurants.)\n3. Take breaks - Its essential to take regular breaks to rest your mind and prevent burnout. Take a walk...go ahead and pace around the house a few times or stretch your legs to refresh your mind.\n4. Use the right tools - Use tools like task lists, time trackers, and project management software to keep yourself organized and on track. (Ask me sometime how I hacked up a secret formula, that took me years of gut wrenching frustration to develop, and has me laughing at “to-do” lists.)\n5. Stay motivated – Keep the fire under your butt. Maintain your ambition. Feed it. Find ways to stay motivated, such as rewarding yourself after completing a task or working with a colleague to stay accountable.\nBy implementing these tips, you can increase your productivity and stay focused throughout the workday.\nIf you want to learn more about ways to boost your productivity at work, shoot me an email at justinmcdonaldusagmail.com. Ill add you to my list and send you exclusive tips and insights to help you become a more curious and successful marketer.},{from:gpt,markdown:None,text:None,value:#ProductivityTips #StayFocused #EntrepreneurMindset #GoalSetting #EliminateDistractions #TakeBreaks #TaskManagement #Motivation #SuccessTips #ProductivityHacks #WorkSmarter #WorkLifeBalance},{from:human,markdown:None,text:None,value:write a Ben Settle style P.S. for the following blog/email, that has a CTA for the Catalyst Code Book that will be coming out this Spring and the reader can reserve a copy and get a FREE digital download one-page cheat sheet (make up a cool name with blueprint in the name): Why Self-Care is Important for Your Mental Health\nIt\s easy to get caught up in the demands of daily life and neglect your mental health.\nHowever, taking care of yourself is essential to your overall well-being. \nSelf-care can include various activities, such as getting enough sleep, exercising, practicing mindfulness, and spending time with loved ones.\nHere are a few reasons why self-care is vital for your mental health:\n1. Reduce stress - Self-care can help you manage stress and prevent burnout. By taking time to relax and recharge, you can reduce the effects of chronic stress on your mental and physical health.\n2. Improve mood - Engaging in self-care activities can improve your mood and promote feelings of happiness and contentment. By focusing on yourself and your needs, you can boost your self-esteem and confidence.\n3. Increase resilience - Regular self-care can help you build resilience and cope with challenges more effectively. By taking care of yourself, you can strengthen your mental and emotional health.\n4. Enhance relationships - Spending time with loved ones is a crucial part of self-care. By nurturing your relationships, you can build a support system that can help you through difficult times.\nBy prioritizing self-care, you can. Take time each day to focus on yourself and your needs.\nIf you want to learn more about ways to improve your mental health and well-being, shoot me an email at justinmcdonaldusagmail.com. I\ll add you to my list and send you exclusive tips and insights to help you become a more curious and successful marketer.},{from:gpt,markdown:None,text:None,value:P.S. Don\t neglect your mental health - it\s the foundation of your success in business and life. And if you\re looking for a powerful catalyst to take your life to the next level, keep an eye out for my upcoming book, The Catalyst Code, coming out this Spring. You can reserve your copy now and get a FREE digital download of my exclusive Blueprint for Mental Wellness one-page cheat sheet. Just shoot me an email at [justinmcdonaldusagmail.com](mailto:justinmcdonaldusagmail.com) and I\ll make sure you\re the first to know when it\s available.},{from:human,markdown:None,text:None,value:produce several twitter tags for that article},{from:gpt,markdown:None,text:None,value:1. #selfcare\n2. #mentalhealth\n3. #wellness\n4. #mindfulness\n5. #resilience\n6. #selflove\n7. #stressmanagement\n8. #selfimprovement\n9. #happiness\n10. #loveyourself\n11. #personaldevelopment\n12. #selfacceptance\n13. #mentalwellness\n14. #mindbodysoul\n15. #healthyhabits},{from:human,markdown:None,text:None,value:list the best 5 and separate with spaces},{from:gpt,markdown:None,text:None,value:#selfcare #mentalhealth #wellness #resilience #mindfulness},{from:human,markdown:None,text:None,value:knowing what you learned from Justin McDonald and Ben Settles email writing style, give me three more infotaining articles to be used for email or blog on Medium.com, and have creative and compelling CTA to email justin at justinmcdonaldusagmail.com that is unique to each article, and have them be a continuation of the last three articles provided, also sprinkle in some lessons, here and there, as if they were coming from Dan Kennedy or Gary Halbert and the top three copywriters of all time},{from:gpt,markdown:None,text:None,value:1. Unlock the Power of Your Mind: How to Master Your Mental State for Success\n* In this article, we\ll dive into the importance of mindset and how you can reprogram your thoughts to achieve success. We\ll explore techniques such as visualization, affirmations, and meditation to help you unlock the power of your mind.\nCTA: Want to learn more about how to master your mental state? Email me at [justinmcdonaldusagmail.com](mailto:justinmcdonaldusagmail.com) and I\ll send you a FREE copy of my Mindset Mastery guide.\n1. The Secret to Writing Compelling Copy: Lessons from the Greatest Copywriters of All Time\n* In this article, we\ll explore the secrets of the greatest copywriters of all time and how you can apply their techniques to your own writing. We\ll discuss the importance of headlines, storytelling, and using emotional triggers to capture your audience\s attention.\nCTA: Ready to take your copywriting to the next level? Email me at [justinmcdonaldusagmail.com](mailto:justinmcdonaldusagmail.com) and I\ll send you a FREE copy of my Copywriting Blueprint cheat sheet.\n1. The Art of Persuasion: How to Sell Anything to Anyone\n* In this article, we\ll dive into the art of persuasion and how you can use it to sell anything to anyone. We\ll discuss techniques such as social proof, scarcity, and the power of storytelling to help you close more deals and make more sales.\nCTA: Want to learn how to become a master persuader? Email me at [justinmcdonaldusagmail.com](mailto:justinmcdonaldusagmail.com) and I\ll send you a FREE copy of my Persuasion Playbook guide.}]]}五、常见变体与注意事项多轮续写部分数据集包含system角色作为首轮用于设定全局指令。元数据扩展有些版本额外增加tools、function_call等字段用于工具调用场景但核心的conversations结构保持不变。清洗细节你使用的ShareGPT_V3_unfiltered_cleaned_small_9k是经过过滤去重、长度裁剪、低质量剔除的子集原始 ShareGPT 数据规模可到 90k 样本对话轮次多达几十轮。注意模板差异加载后需要根据你的模型如 Llama、ChatGLM、Qwen通过tokenizer.apply_chat_template(conversations, tokenizeFalse)转换为对应格式而不是直接拼接value。六、适用场景与局限性适用通用对话 SFT聊天、写作、代码、推理指令微调instruction tuning——将首轮视为指令后续为多轮澄清与回答多轮对话生成评估如 MT-Bench 格式兼容局限不显式标注轮次序号需依赖数组索引无法表达“并行发言”或“分支对话”若缺少角色映射约定如humanvsuser可能造成训练错配七、从 ShareGPT 格式到训练输入一个典型的转换流水线以 Llama 为例defformat_sharegpt(conv_list):# 将 {from: human, value: ...} 映射为 {role: user, content: ...}# 同时保留 system 消息messages[]forturninconv_list:roleuserifturn[from]humanelseassistantifturn[from]system:rolesystemmessages.append({role:role,content:turn[value]})returntokenizer.apply_chat_template(messages,tokenizeFalse)八、总结ShareGPT 格式以其极简性和高适应性成为开源社区对话微调数据的首选结构Alpaca、Vicuna、UltraChat 等均受其影响。只需记住三件事对话列表conversations是灵魂。角色字段from定身份value定内容。模板转换加载后适配目标模型的 chat template 再送入训练。2026-07-21二