Flash attention(Fast and Memory-Efficient Exact Attention with IO-Awareness): A deep dive

Flash attention is power optimization transformer attention mechanism that provides 15% efficiency