[openib-general] Re: RDS RX buf allocation why on RX callback flow?

[Or Gerlitz]

On 4/27/06, Ranjit Pandit <rpandit at silverstorm.com> wrote:
> On 4/27/06, Leonid Arsh <leonida at voltaire.com> wrote:

>> During the run we get error messages in dmesg on the server side.
>> Have you seen anything like this?
>> Please see the dmesg output below:

> I will see if I can reproduce it.

I think the issue here is not to reproduce it (easy)  but to understand/discuss
the design. You are doing GFP_ATOMIC allocation in the rx callback
flow which can
fail ofcourse but since this is hard irq context you can't use GFP_KERNEL.

Since RDS comes to offload Oracle IPC which is somehow transactional
by nature (at least the cache fusions) does it make sense to you in
the server side to post initial rx buffers and then for each TX before
(after) post sending it just post another rx buffer. Same for the
client side before (after) posting tx post an rx. This would make the
rx posting from thread (process) context and you can use GFP_KERNEL.

What about the other tpyes of Oracle IPC, do they also have
transactional (req/resp) nature?

Or.

> > swapper: page allocation failure. order:1, mode:0x20
> >
> > Call Trace: <IRQ> <ffffffff801572ae>{__alloc_pages+662}
> > <ffffffff801184c7>{smp_apic_timer_interrupt+54}
> >       <ffffffff8010e63c>{apic_timer_interrupt+132}
> > <ffffffff8015a0ff>{cache_grow+288}
> >       <ffffffff8015a4ef>{cache_alloc_refill+419}
> > <ffffffff80159fb2>{kmem_cache_alloc+87}
> >       <ffffffff8824c01c>{:ib_rds:rds_alloc_buf+16}
> > <ffffffff8824c0f1>{:ib_rds:rds_alloc_recv_buffer+12}
> >       <ffffffff8824b377>{:ib_rds:rds_post_new_recv+23}
> > <ffffffff8824bfc3>{:ib_rds:rds_recv_completion+85}
> >       <ffffffff88249b6f>{:ib_rds:rds_cq_callback+87}
> > <ffffffff8814882b>{:ib_mthca:mthca_eq_int+119}
> >       <ffffffff801102d8>{do_IRQ+50} <ffffffff8010de1e>{ret_from_intr+0}
> >       <ffffffff88148b45>{:ib_mthca:mthca_tavor_interrupt+91}
> >       <ffffffff80151bd5>{handle_IRQ_event+41}
> > <ffffffff80151ca2>{__do_IRQ+156}
> >       <ffffffff801102d3>{do_IRQ+45} <ffffffff8010de1e>{ret_from_intr+0}
> >        <EOI> <ffffffff8010be87>{mwait_idle+54}
> > <ffffffff8010be37>{cpu_idle+93}
> >       <ffffffff8052733d>{start_secondary+1131}