-
-| High Level APIs | Sample use | Arguments |
-|-----------------|------------|-------------------|
-| QEfficient.cloud.infer | [click here](#1-use-qefficientcloudinfer) |
|
-| QEfficient.cloud.execute | [click here](#2-use-of-qefficientcloudexcute) |
|
-
-**One argument, prompt or prompts_txt_file_path must be passed.**
-
-### 1. Use QEfficient.cloud.infer
-
-This is the single e2e python api in the library, which takes model_card name as input along with other compile args if necessary and does everything in one go.
-
-* Torch Download → Optimize for Cloud AI 100 → Export to ONNX → Verify (CPU) → Compile on Cloud AI 100 → [Execute](#2-use-of-qefficientcloudexecute)
-* It skips the ONNX export/compile stage if ONNX file or qpc found on path
-
-
-```bash
-# Check out the options using the help menu
-python -m QEfficient.cloud.infer --help
-python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 1 --prompt_len 32 --ctx_len 128 --mxfp6 --num_cores 16 --device_group [0] --prompt "My name is" --mos 1 --aic_enable_depth_first
-
-# If executing for batch size>1,
-
-# Either pass input prompts in single string, seperated with pipe (|) symbol". Example below
-
-python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 3 --prompt_len 32 --ctx_len 128 --num_cores 16 --device_group [0] --prompt "My name is|The flat earth
-theory is the belief that|The sun rises from" --mxfp6 --mos 1 --aic_enable_depth_first
-
-# Or pass path of txt file with input prompts, Example below, sample txt file(prompts.txt) is present in examples folder.
-
-python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 3 --prompt_len 32 --ctx_len 128 --num_cores 16 --device_group [0] --prompts_txt_file_path examples/prompts.txt --mxfp6 --mos 1 --aic_enable_depth_first
- ```
-### 2. Use of QEfficient.cloud.execute
-
-Once we have compiled the QPC, we can now use the precompiled QPC in execute API to run for different prompts, like below:
-
+## Quick Installation
```bash
-python -m QEfficient.cloud.execute --model_name gpt2 --qpc_path qeff_models/gpt2/qpc_16cores_1BS_32PL_128CL_1devices_mxfp6/qpcs --prompt "Once upon a time in" --device_group [0]
-```
-
-We can also enable MQ, just based on the number of devices. Based on the "--device_group" as input it will create TS config on the fly. If "--device_group [0,1]" it will create TS config for 2 devices and use it for compilation, if "--device_group 0" then TS compilation is skipped and single soc execution is enabled.
-
-```bash
-python -m QEfficient.cloud.infer --model_name Salesforce/codegen-2B-mono --batch_size 1 --prompt_len 32 --ctx_len 128 --mxfp6 --num_cores 16 --device_group [0,1] --prompt "def fibonacci(n):" --mos 1 --aic_enable_depth_first
-
-# Once qpc is saved, you can use the execute API to run for different prompts
-python -m QEfficient.cloud.execute --model_name Salesforce/codegen-2B-mono --qpc-path qeff_models/Salesforce/codegen-2B-mono/qpc_16cores_1BS_32PL_128CL_2devices_mxfp6/qpcs --prompt "def binary_search(array: np.array, k: int):" --device_group [0,1]
-
-# To disable MQ, just pass single soc like below:
-python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 1 --prompt_len 32 --ctx_len 128 --mxfp6 --num_cores 16 --device_group [0] --prompt "My name is" --mos 1 --aic_enable_depth_first
-```
-
-
-
-
-| High Level APIs | Single SoC | Tensor Slicing |
-|-----------------|------------|-------------------|
-| QEfficient.cloud.infer | python -m QEfficient.cloud.infer --model_name $\color{green} {model}$ --batch_size 1 --prompt_len 128 --ctx_len 1024 --num_cores 16 --device_group [0] --prompt "My name is" --mxfp6 --hf_token $\color{green}{xyz}$ --mos 1 --aic_enable_depth_first | python -m QEfficient.cloud.infer --model_name $\color{green}{model}$ --batch_size 1 --prompt_len 128 --ctx_len 1024 --num_cores 16 --device_group [0,1,2,3] --prompt "My name is" --mxfp6 --hf_token $\color{green}{xyz}$ --mos 1 --aic_enable_depth_first |
-| QEfficient.cloud.execute | python -m QEfficient.cloud.execute --model_name $\color{green}{model}$ --device_group [0] --qpc_path $\color{green}{path}$ --prompt "My name is" --hf_token $\color{green}{xyz}$ | python -m QEfficient.cloud.execute --model_name $\color{green}{model}$ --device_group [0,1,2,3] --qpc_path $\color{green}{path}$ --prompt "My name is" --hf_token $\color{green}{xyz}$ |
-
-:memo: Replace $\color{green}{model}$ , $\color{green}{path}$ and $\color{green}{xyz}$ with preferred model card name, qpc path and hf token respectively.
-
-
-
-## Using Low Level APIs
-
-
-
+# Create Python virtual env and activate it. (Required Python 3.8)
-| Low Level APIs | Sample use | Arguments |
-|-----------------|------------|-------------------|
-| QEfficient.transform | [click here](#1-model-download-and-transform) |
model : $\color{green} {Mandatory}$form_factor : Optional [Default="cloud"] |
-| QEfficient.export | [click here](#2-onnx-export-of-transformed-model) |
model_name : $\color{green} {Mandatory}$model_kv : Optional local_model_dir : Optional [Path to custom model weights and config file] tokenizer : Optional cache_dir : Optional [Path to the directory used for saving HuggingFace cache, Default is "efficient-transformers/cache_dir".] onnx_dir_path : Optional hf_token : Optionalseq_length : Optional [Default=128]kv : Optional [Default=True]form_factor : Optional [Default="cloud"]
-| QEfficient.compile | [click here](#3-compile-on-cloud-ai-100) |
onnx_path : $\color{green} {Mandatory}$qpc_path : $\color{green} {Mandatory}$num_cores : $\color{green} {Mandatory}$device_group : $\color{green} {Mandatory}$ batch_size : Optional [Default=1] prompt_len : Optional [Default=32]ctx_len : Optional [Default=128]aic_enable_depth_first : Optional [Default=False] mos : Optional [Default=-1] mxint8 : Optional [Defaul=False]mxfp6 : Optional [Default=True] custom_io_file_path : Optional [Default=None]|
-|QEfficient.cloud_ai_100_exec_kv | [click here](#4-run-benchmark) |
tokenizer : $\color{green} {Mandatory}$ qpc_path : $\color{green} {Mandatory}$ **prompt : Optional **prompts_txt_file_path : Optional device_id : Optional [Default=[0]] generation_len : Optional [Default=None] enable_debug_logs : Optional [Default=False] stream : Optional [Default=True] write_io_dir : Optionalautomation : Optional [Default=False]|
-
-**In QEfficient.cloud_ai_100_exec_kv, atleast one argument, prompt or prompts_txt_file_path must be passed.
-
-### 1. Model download and Optimize for Cloud AI 100
-
-Initialize QEfficient and transform the models, Check the list of supported architectures in the repo.
-
-```Python
-# Initiate the Orignal Transformer model
-import os
-
-from QEfficient import QEFFAutoModelForCausalLM as AutoModelForCausalLM
-
-# Please uncomment and use appropriate Cache Directory for transformers, in case you don't want to use default ~/.cache dir.
-# os.environ["TRANSFORMERS_CACHE"] = "/local/mnt/workspace/hf_cache"
-
-# ROOT_DIR = os.path.dirname(os.path.abspath(""))
-# CACHE_DIR = os.path.join(ROOT_DIR, "tmp") #, you can use a different location for just one model by passing this param as cache_dir in below API.
-
-# Model-Card name to be onboarded (This is HF Model Card name) : https://huggingface.co/gpt2-xl
-model_name = "gpt2" # Similar, we can change model name and generate corresponding models, if we have added the support in the lib.
-
-qeff_model = AutoModelForCausalLM.from_pretrained(model_name)
-print(f"{model_name} optmized for AI 100 \n", qeff_model)
-```
-
-### 2. Export and Compile with one API
-
-use the qualcomm_efficient_converter API to export the KV transformed Model to ONNX and Verify on Torch.
-
-```Python
-# We can now export the modified models to Onnx framework
-# This will generate single Onnx Model for both Prefill and Decode Variations which are optimized for
-# Cloud AI 100 Platform.
-
-# While generating the ONNX model, this will clip the overflow constants to fp16
-# Verify the model on Onnxruntime vs Pytorch
-
-# Then generate inputs and customio yaml file required for compilation.
-# Compile the model for provided compilation arguments
-# Please use platform SDk to Check num_cores for your card.
-
-generated_qpc_path = qeff_model.compile(
- num_cores=14,
- mxfp6=True,
- device_group=[0],
-)
-```
+python3.8 -m venv qeff_env
+source qeff_env/bin/activate
+pip install -U pip
-### 3. Run Benchmark
+# Clone and Install the QEfficient Repo.
+pip install git+https://github.com/quic/efficient-transformers
-Benchmark the model on Cloud AI 100, run the infer API to print tokens and tok/sec
+```
-```Python
-# post compilation, we can print the latency stats for the kv models, We provide API to print token and Latency stats on AI 100
-# We need the compiled prefill and decode qpc to compute the token generated, This is based on Greedy Sampling Approach
+For more details about using ``QEfficient`` via Cloud AI 100 Apps SDK, visit [Linux Installation Guide](https://quic.github.io/efficient-transformers/source/Linux_installation.html)
-qeff_model.generate(prompts=["My name is"])
-```
-End to End demo examples for various models are available in **notebooks** directory. Please check them out.
-## Adding support for a new model
-Watch this space for references to detailed steps, template examples and much more.
+## Documentation
-## Details on KV Cache Optimization for Cloud AI 100
-
+* [Quick Start Guide](https://quic.github.io/efficient-transformers/source/quick_start.html#)
+* [Python API](https://quic.github.io/efficient-transformers/source/high_level_api.html)
+* [Validated Models](https://quic.github.io/efficient-transformers/source/Validate.html)
+* [Models coming soon](https://quic.github.io/efficient-transformers/source/Validate.html#models-coming-soon)
-Note: More details are here: https://quic.github.io/cloud-ai-sdk-pages/latest/Getting-Started/Model-Architecture-Support/Large-Language-Models/llm/
+> Note: More details are here: https://quic.github.io/cloud-ai-sdk-pages/latest/Getting-Started/Model-Architecture-Support/Large-Language-Models/llm/
## Acknowledgements
Thanks to:
-* Huggingface transformers for work in LLM GenAI modeling implementation
+* HuggingFace transformers for work in LLM GenAI modeling implementation
* ONNX, Pytorch, ONNXruntime community.
## Support
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 000000000..9fe094f98
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,25 @@
+# Docs
+
+This directory contains the instructions for building static html documentations based on [sphinx](https://www.sphinx-doc.org/en/master/).
+
+
+## Build the docs
+Install the packages required for building documentation:
+
+```sh
+ pip install -r docs/requirements.txt
+```
+
+And then, change directory to docs folder to build the docs.
+
+```sh
+cd docs/
+sphinx-build -M html . build
+```
+## Preview the docs locally
+
+```bash
+cd build/html
+python -m http.server
+```
+You can visit the page with your web browser with url `http://localhost:8080`.
\ No newline at end of file
diff --git a/docs/__init__.py b/docs/__init__.py
deleted file mode 100644
index 91fee0a49..000000000
--- a/docs/__init__.py
+++ /dev/null
@@ -1,7 +0,0 @@
-# -----------------------------------------------------------------------------
-#
-# Copyright (c) 2023-2024 Qualcomm Innovation Center, Inc. All rights reserved.
-# SPDX-License-Identifier: BSD-3-Clause
-#
-# -----------------------------------------------------------------------------
-
diff --git a/docs/_static/my_theme.css b/docs/_static/my_theme.css
new file mode 100644
index 000000000..00a18c905
--- /dev/null
+++ b/docs/_static/my_theme.css
@@ -0,0 +1,3 @@
+.wy-nav-content {
+ max-width: 1200px !important;
+}
\ No newline at end of file
diff --git a/docs/conf.py b/docs/conf.py
new file mode 100644
index 000000000..55881fb44
--- /dev/null
+++ b/docs/conf.py
@@ -0,0 +1,72 @@
+# -----------------------------------------------------------------------------
+#
+# Copyright (c) 2024 Qualcomm Innovation Center, Inc. All rights reserved.
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# -----------------------------------------------------------------------------
+
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath(".."))
+
+
+# -- Project information -----------------------------------------------------
+
+project = "efficient-transformers"
+copyright = "2024, Qualcomm"
+
+# The full version, including alpha/beta/rc tags
+release = "main"
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = ["myst_parser", "sphinx.ext.todo", "sphinx.ext.viewcode", "sphinx.ext.autodoc"]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+#
+html_theme = "sphinx_rtd_theme"
+
+
+def setup(app):
+ app.add_css_file("my_theme.css")
+
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]
+source = [".md"]
+todo_include_todos = True
+
+suppress_warnings = [
+ "ref.rst_pilog", # Suppress warnings about excluded toctree entries
+]
diff --git a/docs/Cloud_AI_100.png b/docs/image/Cloud_AI_100.png
similarity index 100%
rename from docs/Cloud_AI_100.png
rename to docs/image/Cloud_AI_100.png
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 000000000..dd4b7ecce
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,69 @@
+% QEfficient documentation master file, created by
+% sphinx-quickstart on Tue May 28 09:19:122024.
+% You can adapt this file completely to your liking, but it should at least
+% contain the root `toctree` directive.
+
+Welcome to Efficient-Transformers Documentation!
+========================================
+
+
+
+
+
+```{toctree}
+:caption: 'Getting Started'
+:maxdepth: 4
+
+source/introduction
+source/validate
+```
+
+
+```{toctree}
+:caption: 'Installation'
+:maxdepth: 2
+
+source/installation
+```
+
+
+```{toctree}
+:caption: 'Quick start'
+:maxdepth: 4
+
+source/quick_start
+```
+
+```{toctree}
+:caption: 'Command Line Interface Use (CLI)'
+:maxdepth: 2
+source/cli_api
+```
+
+
+```{toctree}
+:caption: 'Python API'
+:maxdepth: 2
+
+source/hl_api
+source/ll_api
+
+```
+
+```{toctree}
+:caption: 'Blogs'
+:maxdepth: 2
+
+source/blogs
+
+```
+
+```{toctree}
+:caption: 'Reference'
+:maxdepth: 2
+
+source/reference
+
+```
diff --git a/docs/requirements.txt b/docs/requirements.txt
new file mode 100644
index 000000000..708f4974c
--- /dev/null
+++ b/docs/requirements.txt
@@ -0,0 +1,3 @@
+Sphinx==7.1.2
+sphinx-rtd-theme==2.0.0
+myst-parser==3.0.1
diff --git a/docs/source/blogs.md b/docs/source/blogs.md
new file mode 100644
index 000000000..6e20dd7f2
--- /dev/null
+++ b/docs/source/blogs.md
@@ -0,0 +1,15 @@
+# Train anywhere, Infer on Qualcomm Cloud AI 100
+ [Click here](https://www.qualcomm.com/developer/blog/2024/01/train-anywhere-infer-qualcomm-cloud-ai-100)
+
+# How to Quadruple LLM Decoding Performance with Speculative Decoding (SpD) and Microscaling (MX) Formats on Qualcomm® Cloud AI 100
+ [Click here](https://statics.teams.cdn.office.net/evergreen-assets/safelinks/1/atp-safelinks.html)
+
+# Power-efficient acceleration for large language models – Qualcomm Cloud AI SDK
+ [Click here](https://www.qualcomm.com/developer/blog/2023/11/power-efficient-acceleration-large-language-models-qualcomm-cloud-ai-sdk)
+
+# Qualcomm Cloud AI 100 Accelerates Large Language Model Inference by ~2x Using Microscaling (Mx) Formats
+[click here](https://www.qualcomm.com/developer/blog/2024/01/qualcomm-cloud-ai-100-accelerates-large-language-model-inference-2x-using-microscaling-mx)
+
+# Qualcomm Cloud AI Introduces Efficient Transformers: One API, Infinite Possibilities
+[click here](https://www.qualcomm.com/developer/blog/2024/05/qualcomm-cloud-ai-introduces-efficient-transformers--one-api--in)
+
diff --git a/docs/source/cli_api.md b/docs/source/cli_api.md
new file mode 100644
index 000000000..32f3247b3
--- /dev/null
+++ b/docs/source/cli_api.md
@@ -0,0 +1,26 @@
+
+```{NOTE}
+Use ``bash terminal``, else if using ``ZSH terminal`` then ``device_group``should be in single quotes e.g. ``'--device_group [0]'``
+```
+
+(infer_api)=
+# `QEfficient.cloud.infer`
+```{eval-rst}
+.. automodule:: QEfficient.cloud.infer.main
+```
+# `QEfficient.cloud.execute`
+```{eval-rst}
+.. automodule:: QEfficient.cloud.execute.main
+```
+# `QEfficient.cloud.compile`
+```{eval-rst}
+ .. automodule:: QEfficient.compile.compile_helper.compile
+ .. code-block:: bash
+
+ python -m QEfficient.cloud.compile OPTIONS
+```
+# `QEfficient.cloud.export`
+```{eval-rst}
+ .. automodule:: QEfficient.cloud.export.main
+
+```
\ No newline at end of file
diff --git a/docs/source/hl_api.md b/docs/source/hl_api.md
new file mode 100644
index 000000000..951309711
--- /dev/null
+++ b/docs/source/hl_api.md
@@ -0,0 +1,35 @@
+**This page give you an overview about the all the APIs that you might need to integrate the `QEfficient` into your python applications.**
+
+# High Level API
+
+## `QEFFAutoModelForCausalLM`
+```{eval-rst}
+.. automodule:: QEfficient.transformers.models.modeling_auto
+ :inherited-members:
+ :undoc-members:
+ :exclude-members: QEffAutoModel,QEFFTransformersBase, run_ort, run_pytorch, get_tokenizer, run_cloud_ai_100, execute
+```
+## `export`
+```{eval-rst}
+.. automodule:: QEfficient.exporter.export_hf_to_cloud_ai_100
+ :members:
+ :show-inheritance:
+ :exclude-members: convert_to_cloud_kvstyle, convert_to_cloud_bertstyle
+```
+## `compile`
+```{eval-rst}
+.. automodule:: QEfficient.compile.compile_helper
+ :members:
+ :show-inheritance:
+.. code-block:: bash
+ import QEfficient
+ base_path, onnx_model_path = QEfficient.export(model_name="gpt2")
+ qpc_path = QEfficient.compile(onnx_path=onnx_model_path, qpc_path=os.path.join(base_path, "qpc"), num_cores=14, device_group=[0])
+```
+## `Execute`
+```{eval-rst}
+.. automodule:: QEfficient.generation.text_generation_inference
+ :members:
+ :show-inheritance:
+ :exclude-members: latency_stats_bertstyle,cloud_ai_100_exec_kv_helper
+```
\ No newline at end of file
diff --git a/docs/source/image/Cloud_AI_100.png b/docs/source/image/Cloud_AI_100.png
new file mode 100644
index 000000000..54ab44309
Binary files /dev/null and b/docs/source/image/Cloud_AI_100.png differ
diff --git a/docs/kv_cache_cloudai100.png b/docs/source/image/kv_cache_cloudai100.png
similarity index 100%
rename from docs/kv_cache_cloudai100.png
rename to docs/source/image/kv_cache_cloudai100.png
diff --git a/docs/source/installation.md b/docs/source/installation.md
new file mode 100644
index 000000000..1a08928f4
--- /dev/null
+++ b/docs/source/installation.md
@@ -0,0 +1,44 @@
+# Pre-requisites
+System Requirements:
+1. [Supported Linux OS](https://quic.github.io/cloud-ai-sdk-pages/latest/Getting-Started/Installation/#operating-systems) - Ubuntu, RHEL and AWS Linux
+2. [Cloud AI 100 Platform and Apps SDK installed](https://quic.github.io/cloud-ai-sdk-pages/latest/Getting-Started/Installation/Cloud-AI-SDK/Cloud-AI-SDK/)
+3. [SDK Pre-requisites](https://quic.github.io/cloud-ai-sdk-pages/latest/Getting-Started/Installation/Pre-requisites/pre-requisites/)
+4. [Multi-device support enabled for model sharding](https://github.com/quic/cloud-ai-sdk/tree/1.12/utils/multi-device)
+
+# Linux Installation
+There are two different way to install efficient-transformers.
+
+## Using SDK
+
+* Download Apps SDK: [Cloud AI 100 Platform and Apps SDK install](https://quic.github.io/cloud-ai-sdk-pages/latest/Getting-Started/Installation/Cloud-AI-SDK/Cloud-AI-SDK/)
+
+
+```bash
+# Install using Apps SDK
+
+bash install.sh --enable-qeff
+source /opt/qti-aic/dev/python/qeff/bin/activate
+
+```
+## Using GitHub Repository
+
+```bash
+
+# Create Python virtual env and activate it. (Required Python 3.8)
+
+python3.8 -m venv qeff_env
+source qeff_env/bin/activate
+pip install -U pip
+
+# Clone and Install the QEfficient Repo.
+pip install git+https://github.com/quic/efficient-transformers
+
+```
+
+# Sanity Check
+
+After any of the above installation methods, you can check if ``QEfficient`` is installed correctly by using
+```bash
+python -c "import QEfficient; print(QEfficient.__version__)"
+```
+If the above line executes successfully, you are good to go ahead and start deploying models on ``Cloud AI 100`` cards using ``QEfficient`` library.
diff --git a/docs/source/introduction.md b/docs/source/introduction.md
new file mode 100644
index 000000000..f6a214106
--- /dev/null
+++ b/docs/source/introduction.md
@@ -0,0 +1,33 @@
+
+
+
+# Introduction Qualcomm ``efficient-transformers`` library
+
+**Train anywhere, Infer on Qualcomm Cloud AI with a Developer-centric Toolchain**
+
+This library provides reimplemented blocks of LLMs which are used to make the models functional and highly performant on Qualcomm Cloud AI 100.
+We support wide range of [models](validated_models) architectures, for easy efficient deployment on Cloud AI 100 cards. Users only need to provide model card from HuggingFace or Path to the local model and the library will take care of transforming model to it's efficient implementation for Cloud AI 100.
+
+For other models, there is comprehensive documentation to inspire upon the changes needed and How-To(s).
+
+**Typically for LLMs, the library provides:**
+1. Reimplemented blocks from Transformers which enable efficient on-device retention of intermediate states. read more [here](kv_cache)
+2. Graph transformations to enable execution of key operations in lower precision
+3. Graph transformations to replace some operations to other mathematically equivalent operations that are efficient/supported on HW backend
+4. Handling for underflow and overflows in lower precision
+5. Patcher modules to map weights of original model's operations to updated model's operations
+6. Exporter module to export the model source into a `ONNX` Graph.
+7. Sample example applications and demo notebooks
+8. Unit test templates.
+
+***Latest news*** :
+
+- [coming soon] Support for more popular [models](coming_soon_models) and inference optimization techniques like continuous batching and speculative decoding
+* [06/2024] Added support for [GPT-J-6B](https://huggingface.co/EleutherAI/gpt-j-6b)
+- [06/2024] Added support for [Qwen2-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2-1.5B-Instruct)
+- [06/2024] Added support for [StarCoder2-15B](https://huggingface.co/bigcode/starcoder2-15b)
+- [06/2024] Added support for [Phi3-Mini-4K-Instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct)
+- [06/2024] Added support for [Codestral-22B-v0.1](https://huggingface.co/mistralai/Codestral-22B-v0.1)
+- [06/2024] Added support for [Vicuna-v1.5](https://huggingface.co/lmsys/vicuna-13b-v1.5)
+- [05/2024] Added support for [Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) & [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1).
+- [04/2024] Initial release of [efficient transformers](https://github.com/quic/efficient-transformers) for seamless inference on pre-trained LLMs.
\ No newline at end of file
diff --git a/docs/source/ll_api.md b/docs/source/ll_api.md
new file mode 100644
index 000000000..8cdb974bc
--- /dev/null
+++ b/docs/source/ll_api.md
@@ -0,0 +1,38 @@
+# Low Level API
+
+## `convert_to_cloud_kvstyle`
+```{eval-rst}
+.. automodule:: QEfficient.exporter.export_hf_to_cloud_ai_100
+ :members:
+ :show-inheritance:
+ :exclude-members: qualcomm_efficient_converter, convert_to_cloud_bertstyle
+```
+## `convert_to_cloud_bertstyle`
+```{eval-rst}
+.. automodule:: QEfficient.exporter.export_hf_to_cloud_ai_100
+ :members:
+ :show-inheritance:
+ :exclude-members: qualcomm_efficient_converter, convert_to_cloud_kvstyle
+```
+
+## `utils`
+
+```{eval-rst}
+.. automodule:: QEfficient.utils.device_utils
+ :members:
+ :show-inheritance:
+```
+
+```{eval-rst}
+.. automodule:: QEfficient.utils.generate_inputs
+ :members:
+ :undoc-members:
+ :show-inheritance:
+```
+
+```{eval-rst}
+.. automodule:: QEfficient.utils.run_utils
+ :members:
+ :undoc-members:
+ :show-inheritance:
+```
\ No newline at end of file
diff --git a/docs/source/quick_start.md b/docs/source/quick_start.md
new file mode 100644
index 000000000..cb410c23d
--- /dev/null
+++ b/docs/source/quick_start.md
@@ -0,0 +1,128 @@
+
+QEfficient Library was designed with one goal:
+
+**To make onboarding of models inference straightforward for any Transformer architecture, while leveraging the complete power of Cloud AI platform**
+
+To achieve this, we have 2 levels of APIs, with different levels of abstraction.
+1. Command line interface abstracts away complex details, offering a simpler interface. They're ideal for quick development and prototyping. If you're new to a technology or want to minimize coding effort.
+
+2. Python high level APIs offer more granular control, ideal for when customization is necessary.
+
+## Command Line Interface
+
+```{NOTE}
+Use ``bash terminal``, else if using ``ZSH terminal`` then ``device_group``should be in single quotes e.g. ``'--device_group [0]'``
+```
+
+### QEfficient.cloud.infer
+
+This is the single e2e CLI API, which takes `model_card` name as input along with other compilation arguments. Check [Infer API doc](infer_api) for more details.
+
+* HuggingFace model files Download → Optimize for Cloud AI 100 → Export to `ONNX` → Compile on Cloud AI 100 → [Execute](#qefficientcloudexecute)
+* It skips the export/compile stage based if `ONNX` or `qpc` files are found. If you use infer second time with different compilation arguments, it will automatically skip `ONNX` model creation and directly jump to compile stage.
+
+
+```bash
+# Check out the options using the help
+python -m QEfficient.cloud.infer --help
+python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 1 --prompt_len 32 --ctx_len 128 --mxfp6 --num_cores 16 --device_group [0] --prompt "My name is" --mos 1 --aic_enable_depth_first
+```
+If executing for batch size>1,
+You can pass input prompts in single string but separate with pipe (|) symbol". Example below
+
+```bash
+python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 3 --prompt_len 32 --ctx_len 128 --num_cores 16 --device_group [0] --prompt "My name is|The flat earth
+theory is the belief that|The sun rises from" --mxfp6 --mos 1 --aic_enable_depth_first
+```
+
+You can also pass path of txt file with input prompts when you want to run inference on lot of prompts, Example below, sample txt file(prompts.txt) is present in examples folder.
+
+```bash
+python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 3 --prompt_len 32 --ctx_len 128 --num_cores 16 --device_group [0] --prompts_txt_file_path examples/prompts.txt --mxfp6 --mos 1 --aic_enable_depth_first
+```
+
+### QEfficient.cloud.execute
+You can first run `infer` API and then use `execute` to run the pre-compiled model on Cloud AI 100 cards.
+Once we have compiled the QPC, we can now use the precompiled QPC in execute API to run for different prompts. Make sure to pass same `--device_group` as used during infer. Refer [Execute API doc](execute_api) for more details.
+
+```bash
+python -m QEfficient.cloud.execute --model_name gpt2 --qpc_path qeff_models/gpt2/qpc_16cores_1BS_32PL_128CL_1devices_mxfp6/qpcs --prompt "Once upon a time in" --device_group [0]
+```
+
+### Multi-Qranium Inference
+You can also enable MQ, just based on the number of devices. Based on the `--device-group` as input it will create TS config on the fly. If `--device-group [0,1]` it will create TS config for 2 devices and use it for compilation, if `--device-group [0]` then TS compilation is skipped and single soc execution is enabled.
+
+```bash
+python -m QEfficient.cloud.infer --model_name Salesforce/codegen-2B-mono --batch_size 1 --prompt_len 32 --ctx_len 128 --mxfp6 --num_cores 16 --device-group [0,1] --prompt "def fibonacci(n):" --mos 2 --aic_enable_depth_first
+```
+Above step will save the `qpc` files under `efficient-transformers/qeff_models/{model_card_name}`, you can use the execute API to run for different prompts. This will automatically pick the pre-compiled `qpc` files.
+
+```bash
+python -m QEfficient.cloud.execute --model_name Salesforce/codegen-2B-mono --qpc-path qeff_models/Salesforce/codegen-2B-mono/qpc_16cores_1BS_32PL_128CL_2devices_mxfp6/qpcs --prompt "def binary_search(array: np.array, k: int):" --device-group [0,1]
+```
+
+To disable MQ, just pass single soc like below, below step will compile the model again and reuse the `ONNX` file as only compilation argument are different from above commands.
+
+```bash
+python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 1 --prompt_len 32 --ctx_len 128 --mxfp6 --num_cores 16 --device-group [0] --prompt "My name is" --mos 1 --aic_enable_depth_first
+```
+
+## Python API
+
+### 1. Model download and Optimize for Cloud AI 100
+If your models falls into the model architectures that are [already supported](validated_models), Below steps should work fine.
+Please raise an [issue](https://github.com/quic/efficient-transformers/issues), in case of trouble.
+
+```Python
+# Initiate the Original Transformer model
+# import os
+
+from QEfficient import QEFFAutoModelForCausalLM as AutoModelForCausalLM
+
+# Please uncomment and use appropriate Cache Directory for transformers, in case you don't want to use default ~/.cache dir.
+# os.environ["TRANSFORMERS_CACHE"] = "/local/mnt/workspace/hf_cache"
+
+# ROOT_DIR = os.path.dirname(os.path.abspath(""))
+# CACHE_DIR = os.path.join(ROOT_DIR, "tmp") #, you can use a different location for just one model by passing this param as cache_dir in below API.
+
+# Model-Card name (This is HF Model Card name) : https://huggingface.co/gpt2-xl
+model_name = "gpt2" # Similar, we can change model name and generate corresponding models, if we have added the support in the lib.
+
+qeff_model = AutoModelForCausalLM.from_pretrained(model_name)
+print(f"{model_name} optimized for AI 100 \n", qeff_model)
+```
+
+### 2. Export and Compile with one API
+
+Use the qualcomm_efficient_converter API to export the KV transformed Model to ONNX and Verify on Torch.
+
+```Python
+# We can now export the modified models to ONNX framework
+# This will generate single ONNX Model for both Prefill and Decode Variations which are optimized for
+# Cloud AI 100 Platform.
+
+# While generating the ONNX model, this will clip the overflow constants to fp16
+# Verify the model on ONNXRuntime vs Pytorch
+
+# Then generate inputs and customio yaml file required for compilation.
+# Compile the model for provided compilation arguments
+# Please use platform SDk to Check num_cores for your card.
+
+generated_qpc_path = qeff_model.compile(
+ num_cores=14,
+ mxfp6=True,
+ device_group=[0],
+)
+```
+
+### 3. Execute
+
+Benchmark the model on Cloud AI 100, run the infer API to print tokens and tok/sec
+
+```Python
+# post compilation, we can print the latency stats for the kv models, We provide API to print token and Latency stats on AI 100
+# We need the compiled prefill and decode qpc to compute the token generated, This is based on Greedy Sampling Approach
+
+qeff_model.generate(prompts=["My name is"])
+```
+End to End demo examples for various models are available in **notebooks** directory. Please check them out.
diff --git a/docs/source/reference.md b/docs/source/reference.md
new file mode 100644
index 000000000..37d1ddadc
--- /dev/null
+++ b/docs/source/reference.md
@@ -0,0 +1,6 @@
+**References**
+# [Qualcomm Cloud AI home](https://www.qualcomm.com/products/technology/processors/cloud-artificial-intelligence)
+# [Qualcomm Cloud AI SDK download](https://www.qualcomm.com/products/technology/processors/cloud-artificial-intelligence/cloud-ai-100#Software)
+# [Qualcomm Cloud AI API reference](https://quic.github.io/cloud-ai-sdk-pages/latest/API/)
+# [User Guide](https://quic.github.io/cloud-ai-sdk-pages/)
+# [OCP Microscaling Formats (MX) Specification](https://www.qualcomm.com/developer/blog/2024/05/6.%09https:/www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf)
\ No newline at end of file
diff --git a/docs/source/validate.md b/docs/source/validate.md
new file mode 100644
index 000000000..81d1fb9b8
--- /dev/null
+++ b/docs/source/validate.md
@@ -0,0 +1,34 @@
+(validated_models)=
+# Validated Models
+
+* [GPT2](https://huggingface.co/openai-community/gpt2)
+* [Llama-3-8b](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
+* [Llama-3-70b](https://huggingface.co/meta-llama/Meta-Llama-3-70B)
+* [Llama-2-70b](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf)
+* [Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)
+* [Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf)
+* [CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf)
+* [CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf)
+* [CodeLlama-34b-hf](https://huggingface.co/codellama/CodeLlama-34b-hf)
+* [Salesforce/codegen25-7b-mono_P](https://huggingface.co/Salesforce/codegen25-7b-mono_P)
+* [Salesforce/xgen-7b-8k-base](https://huggingface.co/Salesforce/xgen-7b-8k-base)
+* [MPT-7b](https://huggingface.co/mosaicml/mpt-7b)
+* [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)
+* [Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)
+* [Vicuna-v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0)
+* [Vicuna-v1.3](https://huggingface.co/lmsys/vicuna-13b-v1.3)
+* [Vicuna-v1.5](https://huggingface.co/lmsys/vicuna-13b-v1.5)
+* [Qwen2-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2-1.5B-Instruct)
+* [StarCoder2-15B](https://huggingface.co/bigcode/starcoder2-15b)
+* [Phi3-Mini-4K-Instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct)
+* [Codestral-22B-v0.1](https://huggingface.co/mistralai/Codestral-22B-v0.1)
+* [Falcon-40b](https://huggingface.co/tiiuae/falcon-40b)
+* [GPT-J-6B](https://huggingface.co/EleutherAI/gpt-j-6b)
+
+(coming_soon_models)=
+# Models Coming Soon
+
+* [Jais-13b](https://huggingface.co/core42/jais-13b)
+* [Jais-30b](https://huggingface.co/core42/jais-30b-chat-v1)
+* [Chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+* [Baichuan2-7B-Base](https://huggingface.co/baichuan-inc/Baichuan2-7B-Base)
\ No newline at end of file
diff --git a/docs/validation.md b/docs/validation.md
deleted file mode 100644
index 7de5b0865..000000000
--- a/docs/validation.md
+++ /dev/null
@@ -1,29 +0,0 @@
-## Validation of Model before and After KV Cache Optimization
-
-# Run APIs
-
-| API |
params | Usage |
-|------------------|:--------:|:--------------------:|
-| run_hf_model_on_pytorch |
- model_hf
|
Runs HuggingFace model on Pytorch and returns output tokens |
-| run_kv_model_on_pytorch |
- model_kv
- n_layer
- padding_shape
|
Runs KV cache model on PyTorch and returns output tokens |
-| run_kv_model_on_ort |
- model_path
- n_layer
- padding_shape
|
Runs KV cache model on onnxruntime and returns output tokens |
-| run_kv_model_on_cloud_ai_100 |
- qpc_path
- device_id
- n_layer
- padding_shape
|
To be run only when device is available runs compiled qpc on Cloud AI 100 and returns output tokens |
-
-Class InputHandler is called inside these APIs which generates inputs for prefill and decode stage.
-
-These APIs, input_generation, checking available device id and Constants scripts are present inside tests/utils folder.
-
-# Sample Usage :
-
-These APIs are part of ApiRunner class :
-
- n_layer : number of layers in the model
- padding_shape : shape of past key values to initialize the first iteration key value inputs with zeros.
-
-
run_api = ApiRunner(tokenizer, Constants.INPUT_STRING, Constants.PROMPT_LEN, Constants.CTX_LEN)
-
pytorch_hf_tokens = run_api.run_hf_model_on_pytorch(model_hf)
-
pytorch_kv_tokens = run_api.run_kv_model_on_pytorch(model_kv, n_layer, padding_shape)
-
ort_tokens = run_api.run_kv_model_on_ort(onnx_model_path, n_layer, padding_shape)
-
cloud_ai_100_tokens = run_api.run_kv_model_on_cloud_ai_100(session, n_layer, padding_shape)
-
-Output tokens are compared to validate the model on the required framework level (PyTorch/ONNX/cloud_ai_100).
diff --git a/examples/__init__.py b/examples/__init__.py
index 91fee0a49..da26921c5 100644
--- a/examples/__init__.py
+++ b/examples/__init__.py
@@ -1,6 +1,6 @@
# -----------------------------------------------------------------------------
#
-# Copyright (c) 2023-2024 Qualcomm Innovation Center, Inc. All rights reserved.
+# Copyright (c) 2024 Qualcomm Innovation Center, Inc. All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause
#
# -----------------------------------------------------------------------------
diff --git a/notebooks/__init__.py b/notebooks/__init__.py
index 91fee0a49..da26921c5 100644
--- a/notebooks/__init__.py
+++ b/notebooks/__init__.py
@@ -1,6 +1,6 @@
# -----------------------------------------------------------------------------
#
-# Copyright (c) 2023-2024 Qualcomm Innovation Center, Inc. All rights reserved.
+# Copyright (c) 2024 Qualcomm Innovation Center, Inc. All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause
#
# -----------------------------------------------------------------------------
diff --git a/pyproject.toml b/pyproject.toml
index ed6619d48..a68318ec9 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -36,6 +36,7 @@ dependencies = [
[project.optional-dependencies]
test = ["pytest","pytest-mock"]
+docs = ["Sphinx==7.1.2","sphinx-rtd-theme==2.0.0","myst-parser==3.0.1"]
quality = ["black", "ruff", "hf_doc_builder@git+https://github.com/huggingface/doc-builder.git"]
[build-system]
diff --git a/scripts/__init__.py b/scripts/__init__.py
index 91fee0a49..da26921c5 100644
--- a/scripts/__init__.py
+++ b/scripts/__init__.py
@@ -1,6 +1,6 @@
# -----------------------------------------------------------------------------
#
-# Copyright (c) 2023-2024 Qualcomm Innovation Center, Inc. All rights reserved.
+# Copyright (c) 2024 Qualcomm Innovation Center, Inc. All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause
#
# -----------------------------------------------------------------------------
diff --git a/tests/__init__.py b/tests/__init__.py
index 91fee0a49..da26921c5 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -1,6 +1,6 @@
# -----------------------------------------------------------------------------
#
-# Copyright (c) 2023-2024 Qualcomm Innovation Center, Inc. All rights reserved.
+# Copyright (c) 2024 Qualcomm Innovation Center, Inc. All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause
#
# -----------------------------------------------------------------------------
-