<?xml version="1.0" encoding="utf-8"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:hal="http://hal.archives-ouvertes.fr/" xmlns:gml="http://www.opengis.net/gml/3.3/" xmlns:gmlce="http://www.opengis.net/gml/3.3/ce" version="1.1" xsi:schemaLocation="http://www.tei-c.org/ns/1.0 http://api.archives-ouvertes.fr/documents/aofr-sword.xsd">
  <teiHeader>
    <fileDesc>
      <titleStmt>
        <title>HAL TEI export of hal-04644928v1</title>
      </titleStmt>
      <publicationStmt>
        <distributor>CCSD</distributor>
        <availability status="restricted">
          <licence target="https://creativecommons.org/publicdomain/zero/1.0/">CC0 1.0 - Universal</licence>
        </availability>
        <date when="2026-05-09T13:50:22+02:00"/>
      </publicationStmt>
      <sourceDesc>
        <p part="N">HAL API Platform</p>
      </sourceDesc>
    </fileDesc>
  </teiHeader>
  <text>
    <body>
      <listBibl>
        <biblFull>
          <titleStmt>
            <title xml:lang="en">Accelerating Large Language Model Inference with Self-Supervised Early Exits</title>
            <author role="aut">
              <persName>
                <forename type="first">Florian</forename>
                <surname>Valade</surname>
              </persName>
              <idno type="halauthorid">3041943-0</idno>
              <affiliation ref="#struct-580722"/>
              <affiliation ref="#struct-307502"/>
            </author>
            <editor role="depositor">
              <persName>
                <forename>florian</forename>
                <surname>valade</surname>
              </persName>
              <email type="md5">9585e26c0d39682108707697826cc6e6</email>
              <email type="domain">univ-eiffel.fr</email>
            </editor>
          </titleStmt>
          <editionStmt>
            <edition n="v1" type="current">
              <date type="whenSubmitted">2024-07-17 10:38:15</date>
              <date type="whenModified">2024-07-23 03:17:28</date>
              <date type="whenReleased">2024-07-18 09:44:15</date>
              <date type="whenProduced">2024-07-17</date>
              <date type="whenEndEmbargoed">2024-07-16</date>
              <ref type="file" target="https://hal.science/hal-04644928v1/document">
                <date notBefore="2024-07-16"/>
              </ref>
              <ref type="file" subtype="author" n="1" target="https://hal.science/hal-04644928v1/file/LLM_Early_exit-5.pdf" id="file-4644928-4037575">
                <date notBefore="2024-07-16"/>
              </ref>
            </edition>
            <edition n="v2">
              <date type="whenSubmitted">2024-07-19 15:53:36</date>
            </edition>
            <edition n="v3">
              <date type="whenSubmitted">2024-07-29 10:41:02</date>
            </edition>
            <edition n="v4">
              <date type="whenSubmitted">2026-02-11 12:03:49</date>
            </edition>
            <respStmt>
              <resp>contributor</resp>
              <name key="1425113">
                <persName>
                  <forename>florian</forename>
                  <surname>valade</surname>
                </persName>
                <email type="md5">9585e26c0d39682108707697826cc6e6</email>
                <email type="domain">univ-eiffel.fr</email>
              </name>
            </respStmt>
          </editionStmt>
          <publicationStmt>
            <distributor>CCSD</distributor>
            <idno type="halId">hal-04644928</idno>
            <idno type="halUri">https://hal.science/hal-04644928</idno>
            <idno type="halBibtex">valade:hal-04644928</idno>
            <idno type="halRefHtml">2024</idno>
            <idno type="halRef">2024</idno>
            <availability status="restricted">
              <licence target="https://about.hal.science/hal-authorisation-v1/">HAL Authorization<ref corresp="#file-4644928-4037575"/></licence>
            </availability>
          </publicationStmt>
          <seriesStmt/>
          <notesStmt/>
          <sourceDesc>
            <biblStruct>
              <analytic>
                <title xml:lang="en">Accelerating Large Language Model Inference with Self-Supervised Early Exits</title>
                <author role="aut">
                  <persName>
                    <forename type="first">Florian</forename>
                    <surname>Valade</surname>
                  </persName>
                  <idno type="halauthorid">3041943-0</idno>
                  <affiliation ref="#struct-580722"/>
                  <affiliation ref="#struct-307502"/>
                </author>
              </analytic>
              <monogr>
                <imprint>
                  <date type="datePub">2024-07-16</date>
                </imprint>
              </monogr>
            </biblStruct>
          </sourceDesc>
          <profileDesc>
            <langUsage>
              <language ident="en">English</language>
            </langUsage>
            <textClass>
              <keywords scheme="author">
                <term xml:lang="en">Deep learning</term>
                <term xml:lang="en">LLM</term>
                <term xml:lang="en">Early Exit</term>
              </keywords>
              <classCode scheme="halDomain" n="stat.ml">Statistics [stat]/Machine Learning [stat.ML]</classCode>
              <classCode scheme="halTypology" n="UNDEFINED">Preprints, Working Papers, ...</classCode>
              <classCode scheme="halOldTypology" n="UNDEFINED">Preprints, Working Papers, ...</classCode>
              <classCode scheme="halTreeTypology" n="UNDEFINED">Preprints, Working Papers, ...</classCode>
            </textClass>
            <abstract xml:lang="en">
              <p>This paper presents a novel technique for accelerating inference in large, pre-trained language models (LLMs) by introducing early exits during inference. The computational demands of these models, used across a wide range of applications, can be substantial. By capitalizing on the inherent variability in token complexity, our approach enables selective acceleration of the inference process. Specifically, we propose the integration of early exit "heads" atop existing transformer layers, which facilitate conditional terminations based on a confidence metric. These heads are trained in a self-supervised manner using the model’s own predictions as training data, thereby eliminating the need for additional annotated data. The confidence metric, established using a calibration set, ensures a desired level of accuracy while enabling early termination when confidence exceeds a predetermined threshold. Notably, our method preserves the original accuracy and reduces computational time on certain tasks, leveraging the existing knowledge of pre-trained LLMs without requiring extensive retraining. This lightweight, modular modification has the potential to greatly enhance the practical usability of LLMs, particularly in applications like real-time language processing in resource-constrained environments.</p>
            </abstract>
          </profileDesc>
        </biblFull>
      </listBibl>
    </body>
    <back>
      <listOrg type="structures">
        <org type="institution" xml:id="struct-580722" status="VALID">
          <idno type="ROR">https://ror.org/03x42jk29</idno>
          <orgName>Université Gustave Eiffel</orgName>
          <date type="start">2020-01-01</date>
          <desc>
            <address>
              <addrLine>Cité Descartes, 5 Boulevard Descartes • Champs-sur-Marne, 77454 Marne-la-Vallée Cedex 2</addrLine>
              <country key="FR"/>
            </address>
            <ref type="url">https://www.univ-gustave-eiffel.fr/</ref>
          </desc>
        </org>
        <org type="institution" xml:id="struct-307502" status="INCOMING">
          <orgName>Fujitsu</orgName>
          <desc>
            <address>
              <country key="FR"/>
            </address>
          </desc>
        </org>
      </listOrg>
    </back>
  </text>
</TEI>